Methods for generating composite images of a patient

ABSTRACT

Methods and systems for generating composite images of a patient and models of the patient&#39;s dentition are provided. In some embodiments, a method includes generating a plurality of 3D models, each comprising a virtual representation of the patient&#39;s dentition at different treatment stages of a treatment plan and obtaining an image of the patient&#39;s face and dentition. The method can include projecting the obtained image into a 3D space to create a projected 3D model of the patient&#39;s face and dentition. The method can also include generating a plurality of modified images, each representing the patient in a treatment stage of the treatment plan and formed by combining the patient&#39;s dentition of one of the 3D models with the patient&#39;s face of the projected 3D model. A modified image corresponding to a selected treatment stage and a corresponding 3D model can be provided for display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/455,441, filed Jun. 27, 2019, which claims the benefit of U.S.Provisional Application No. 62/692,548, filed Jun. 29, 2018, each ofwhich is herein incorporated by reference in its entirety.

BACKGROUND

In the design of virtual representations of living beings, an “uncannyvalley” can relate to the extent a virtual object's resemblance to aliving being corresponds to emotional responses to the virtual object.The concept of an uncanny valley may suggest humanoid virtual objectswhich appear almost, but not exactly, like real human beings (robots, 3Dcomputer animations, lifelike dolls, etc.) elicit uncanny, or strangelyfamiliar, feelings of eeriness and revulsion in observers. A virtualobject that appears “almost” human risks may elicit cold, eerie, and/orother non-emotional feelings in viewers.

In the context of treatment planning, the uncanny valley problem maycause people to negatively react to humanoid representations ofthemselves. As an example, people viewing a 3D virtual representation ofthemselves after an orthodontic treatment plan may be confronted with anunfamiliar, robotic, or non-humanoid view of themselves.

These issues may undermine perceptions of treatment planning proposalsand/or lead to negative perceptions of treatment planning proposals.These issues may influence decision making with respect to the properalignment of teeth in a human patient when three-dimensional (3D)renderings of digital models of patient's teeth are combined withtwo-dimensional (2D) images or photos of a patient.

Furthermore, orthodontic treatment planning processes may not take intoaccount facial relationships between the positions and orientations ofteeth and the shape and position of facial features of a patient.

Systems and methods that reduce the uncanny valley reaction and takeinto account the relationships between the positions and orientations ofteeth and the shape and position of facial features of a patient couldhelp in increasing the effectiveness and acceptance of orthodontictreatment, particularly orthodontic treatments involving virtualrepresentations and/or virtual 3D models of living beings before, duringand/or after the application of orthodontic treatment plans.

SUMMARY

This disclosure generally relates to system and methods of correctingmalocclusions of teeth. More particularly, the disclosure relates tosystem and methods of accurately and realistically depicting 3D bitemodels of teeth in a 2D image of patient and systems and methods ofdetermining the final orthodontic and restorative object positions forteeth.

Systems and methods are described herein to more closely integrate 3Dbite models into 2D images of a patient to aid in reducing oreliminating the uncanny valley reaction and aid in providing betterdecision making with respect to the proper alignment of teeth. Inaddition, orthodontic systems and methods are introduced that evaluatethe shape of a patient's face and the relationships between thepositions and orientations of teeth and the shape and position of facialfeatures of a patient in the treatment planning processes such that thefinal orthodontic and resorted position and shape of the patient teethmore closely match an ideal position with respect to each particularpatient.

A method of generating a virtual depiction of an orthodontic treatmentof a patient is disclosed. The method can be performed as acomputer-implemented method. The method may include building a pluralityof 3D models of the patient's dentition and obtaining an image of thepatient including at least a portion of the patient's face and includingat least a portion of the patient's dentition. The plurality of 3Dmodels can represent the patient's dentition at each of a plurality ofdifferent, specific stages of an orthodontic treatment plan. The methodmay also include selecting a first set of reference points on one of the3D models of the patient's dentition. The method may include selecting asecond set of reference points on the dentition of the image of thepatient and combining the 3D model of the patient's dentition with theimage of the patient. The method may also include aligning the first setof reference points on the 3D model of the patient's dentition with thesecond set of reference points on the image of the patient. The methodmay also include generating a first plurality of modified images of thepatient based on the alignment, each representing the patient in arespective stage of the first orthodontic treatment plan. The method mayalso include generating a second plurality of modified images of thepatient based on the alignment, each representing the patient in arespective stage of the second orthodontic treatment plan. The methodmay also include sending instructions to display at least one of themodified images.

Computer-readable media, computer systems having processors and memory,and computer-implemented methods of orthodontically treating a patientmay comprise: building a three-dimensional model of the patient'sdentition; forming an image of the patient including at least a portionof the patient's face and including at least a portion of the patient'sdentition; selecting a first set of reference points on thethree-dimensional model of the patient's dentition; selecting a secondset of reference points on the dentition of the image of the patient;combining the three-dimensional model of the patient's dentition withthe image of the patient; and aligning the first set of reference pointson the three-dimensional model of the patient's dentition with thesecond set of reference points on the image of the patient.

In some implementations, at least one reference point in the first setof reference points corresponds to a respective reference point in thesecond set of reference points. In some implementations, thecorresponding reference points are at similar locations on thethree-dimensional model and the image of the patient. In someimplementations, the similar locations are the gingival apex of at leasttwo teeth. In some implementations, the similar locations are themidpoint of incisal edges. In some implementations, the similarlocations are cusp tips.

In some implementations, the image of the patient is a two-dimensionalimage of the patient. In some implementations, contours of the teeth inthe three-dimensional model of the patient's dentition are aligned withcontours of teeth in the image of the patient. In some implementations,the aligning includes minimizing the square of the distance between thefirst set of reference points on the three-dimensional model and thesecond set of reference points on the image of the patient.

In some implementations, the method includes showing on the display adisplayed modified image corresponding to the at least one of theplurality of modified images. The method can include providing aninterface that allows a user to select a modified image to display. Themethod can include receiving from the interface an indication of theselected modified image to display, and changing the displayed modifiedimage to the selected modified image in response to the indication. Themethod can further comprise displaying the image of the patient adjacentto the displayed modified image.

In some implementations, sending instructions to display at least one ofthe plurality of modified images includes sending instructions tosimultaneously display a three-dimensional model corresponding to the atleast one image's treatment stage. In some implementations, the modifiedimage and the three-dimensional model are displayed adjacent to oneanother. In some implementations, the displayed three dimensional modelincludes one or more labels indicating a degree of malocclusion for oneor more malocclusions identified in the patient's dentition. In someimplementations, the displayed three dimensional model includes one ormore labels indicating an amount of orthodontic correction to be appliedto parts of the patient's dentition.

In some implementations, the method further comprises transmitting amodified image of the plurality of modified images to the patient in areminder email for an appointment. The modified image can contain aprojected representation of the patient's face including teeth of thepatient's dentition at a selected treatment stage of an orthodontictreatment plan. In some implementations, the selected treatment stage isa final stage of the treatment plan.

In some implementations, the method further comprises transmitting avideo file to the patient. The video file can include representations ofthe plurality of modified images arranged in order to show a projectionof how part or all of the patient's face including part or all of thepatient's dentition will change throughout the treatment plan.

A method of orthodontically generating a virtual depiction of aplurality of alternative orthodontic treatments for a patient isdisclosed. The method can be performed as a computer-implemented method.The method may include building a first plurality and a second pluralityof 3D models of the patient's dentition and obtaining an image of thepatient including at least a portion of the patient's face and includingat least a portion of the patient's dentition. The first plurality of 3Dmodels can represent the patient's dentition at each of a plurality ofdifferent, specific stages of a first orthodontic treatment plan, andthe second plurality of 3D models can represent the patient's dentitionat each of a plurality of different, specific stages of a secondorthodontic treatment plan different from the first plan. The method mayalso include selecting a first set of reference points on one of the 3Dmodels of the patient's dentition. The 3D model can be a model common tothe first and second pluralities of 3D models. The method may includeselecting a second set of reference points on the dentition of the imageof the patient and combining the 3D model of the patient's dentitionwith the image of the patient. The method may also include aligning thefirst set of reference points on the 3D model of the patient's dentitionwith the second set of reference points on the image of the patient. Themethod may also include generating a plurality of modified images of thepatient based on the alignment, each representing the patient in arespective stage of the orthodontic treatment plan. The method may alsoinclude sending instructions to display at least one modified image fromeach of the first and second pluralities of modified images on adisplay.

In some implementations, the first three-dimensional model is a model ofa starting point of the treatment plan, and the image of the patient isan image at the beginning of treatment of the patient.

In some implementations, the methods disclosed herein can be performedas part of the orthodontic treatment of the patient. In someimplementations, instructions to design or manufacture one or moreorthodontic appliance according to the treatment plan are provided.

A system is provided. The system comprises memory storingcomputer-program instructions and one or more physical processorscoupled to the memory. The one or more physical processors areconfigured to implement any one of the computer-implemented methodsdisclosed herein using the computer-program instructions.

A computer-readable medium is provided. The computer-readable mediumcomprises computer-program instructions that, when executed by aprocessor, perform any one of the computer-implemented methods disclosedherein.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIG. 1A illustrates a tooth repositioning appliance, in accordance withone or more embodiments herein;

FIG. 1B illustrates a tooth repositioning system, in accordance with oneor more embodiments herein;

FIG. 1C illustrates a method of orthodontic treatment using a pluralityof appliances, in accordance with one or more embodiments herein;

FIG. 2 illustrates a method for designing an orthodontic appliance, inaccordance with one or more embodiments herein;

FIG. 3 illustrates a method for planning an orthodontic treatment, inaccordance with one or more embodiments herein;

FIG. 4 is a simplified block diagram of a system for designing anorthodontic appliance and planning an orthodontic treatment, inaccordance with one or more embodiments herein;

FIG. 5A depicts a method of building a composite image, in accordancewith one or more embodiments herein;

FIG. 5B depicts a method of generating a virtual depiction of anorthodontic treatment of a patient, in accordance with one or moreembodiments herein;

FIG. 5C depicts a method of generating virtual depictions of a pluralityof alternative orthodontic treatments for a patient, according to one ormore embodiments disclosed herein;

FIG. 6 depicts a two-dimensional (2D or 2-D) image of a patient, inaccordance with one or more embodiments herein;

FIG. 7 depicts a three-dimensional (3D or 3-D) bite model of a patient'steeth, in accordance with one or more embodiments herein;

FIG. 8 depicts the selection of reference points on the 2D image of apatent, in accordance with one or more embodiments herein;

FIG. 9 depicts the selection of reference points on the 3D bite model ofthe patient's teeth, in accordance with one or more embodiments herein;

FIG. 10 depicts the integration of the 3D bite model into the 2D imageof the patient to create a composite image, in accordance with one ormore embodiments herein;

FIG. 11 depicts a close up of a composite image of the 3D bite model andthe 2D image of the patient with overlaid contours, in accordance withone or more embodiments herein;

FIG. 12 depicts a composite image of the 3D bite model in a finalposition and the 2D image of the patient, in accordance with one or moreembodiments herein;

FIG. 13 depicts a composite image of the 3D bite model and the 2D imageof the patient, in accordance with one or more embodiments herein;

FIG. 13A depicts a method for extracting mouth and teeth contours from apatient's two-dimensional image and aligning with contours of a 3D modelof the patients teeth, in accordance with one or more embodimentsherein;

FIG. 13B depicts a method for determine facial landmarks, in accordancewith one or more embodiments herein;

FIG. 13C depicts a method of determining dental contours, in accordancewith one or more embodiments herein;

FIG. 13D depicts a method of determining lip contours and mouthopenings, in accordance with one or more embodiments herein;

FIG. 13E depicts alignment of dental and lip contours with a 2D image ofa patient, in accordance with one or more embodiments herein;

FIG. 13F depicts a morphing process of a facial image of patient betweena pre-treatment state and a predicted post treatment state, inaccordance with one or more embodiments herein;

FIG. 14 depicts the selection of reference points on the 2D image of apatient, in accordance with one or more embodiments herein;

FIG. 15 depicts the selection of reference points on the 3D bite modelof the patient's teeth, in accordance with one or more embodimentsherein;

FIG. 16A depicts a method for planning the treatment of a patient'steeth, in accordance with one or more embodiments herein;

FIG. 16B depicts a method for virtually representing a treatment outcomeusing automated detection of facial and dental reference objects,according to some embodiments.

FIG. 17 depicts a method of selecting and determining reference pointsand lines on an image of a patient, in accordance with one or moreembodiments herein;

FIG. 18 depicts a method of selecting and determining reference pointsand lines on an image of a patient, in accordance with one or moreembodiments herein;

FIG. 19 depicts a method of selecting and determining reference pointsand lines on an image of a patient, in accordance with one or moreembodiments herein;

FIG. 20 depicts a method of selecting and determining reference pointsand lines on an image of a patient, in accordance with one or moreembodiments herein;

FIG. 21 depicts a method of selecting and determining reference pointsand lines on a 3D bite model and an image of a patient, in accordancewith one or more embodiments herein;

FIG. 22 depicts a method of selecting and determining reference pointsand lines on an image of a patient, in accordance with one or moreembodiments herein;

FIG. 23 depicts a method determining face and head shape of a patient,in accordance with one or more embodiments herein;

FIG. 23A depicts a method of aligning a 3D model of a patient's teethwith a facial image of the patient, in accordance with one or moreembodiments herein;

FIG. 23B depicts facial landmarks from a facial image of a patient, inaccordance with one or more embodiments herein;

FIG. 23C shows a process of determining a facial midline from a facialimage of a patient, in accordance with one or more embodiments herein;

FIG. 23D shows a process of aligning a 3D model of a patient's teethwith a facial midline, in accordance with one or more embodimentsherein;

FIG. 24 depicts a method of selecting and determining reference pointsand lines on a 3D bite model of a patient, in accordance with one ormore embodiments herein;

FIG. 25 depicts various possible shapes of teeth of a patient, inaccordance with one or more embodiments herein;

FIG. 26 depicts a method determining a gingival line of a patient, inaccordance with one or more embodiments herein;

FIG. 27 depicts the placement of restorative objects in a 3D bite modelof a patient, in accordance with one or more embodiments herein;

FIG. 28 depicts the determination of the smile arc of a patient, inaccordance with one or more embodiments herein;

FIG. 29 depicts a method of measuring tooth inclination of a patient, inaccordance with one or more embodiments herein;

FIG. 30 depicts a method of determining the target final position of apatient's teeth, in accordance with one or more embodiments herein;

FIG. 31 depicts a 3D bite model integrated into a facial image a targetfinal position, in accordance with one or more embodiments herein;

FIG. 32 illustrates a composite patient image, in accordance with one ormore embodiments herein;

FIG. 33 illustrates a controller for adjusting the features of an image,in accordance with one or more embodiments herein;

FIG. 34 illustrates a 3D bite model in the composite view, in accordancewith one or more embodiments herein;

FIG. 35 illustrates a method of matching the 3D bite model with areference image, in accordance with one or more embodiments herein;

FIG. 36 illustrates a method of matching the 3D bite model with areference image, in accordance with one or more embodiments herein;

FIG. 36A depicts a process of rendering a realistic composite image ofpatient's face and a model of a patient's teeth, in accordance with oneor more embodiments herein;

FIG. 36B depicts a process of rendering a realistic composite image ofpatient's face and a model of a patient's teeth, in accordance with oneor more embodiments herein;

FIG. 37 illustrates the tooth mass reduction for restorative objects, inaccordance with one or more embodiments herein;

FIG. 38 depicts a method of determining of tooth mass low reduction, inaccordance with one or more embodiments herein;

FIG. 39 depicts a 3D bite model with restorative objects, in accordancewith one or more embodiments herein;

FIG. 40 depicts a method of forming a composite image, in accordancewith one or more embodiments herein;

FIG. 41 depicts a method for forming a composite image, in accordancewith one or more embodiments herein;

FIG. 42 depicts a portion of a method for determining tooth dimensions,in accordance with one or more embodiments herein;

FIG. 43 depicts a portion of a method for determining tooth dimensions,in accordance with one or more embodiments herein; and

FIG. 44 depicts a portion of a method for determining tooth dimensions,in accordance with one or more embodiments herein;

FIG. 45 illustrates a virtual depiction 4500 of an orthodontic treatmentof a patient, in accordance with one or more embodiments herein;

FIG. 46 illustrates a comparative display showing a virtual depiction incomparison to an initial image of the patient, in accordance with one ormore embodiments herein;

FIG. 47 illustrates an augmented display showing a modified image of thepatient in conjunction with a 3D model of the patient's dentition, inaccordance with one or more embodiments herein; and

FIG. 48 illustrates a treatment-comparison display 4800 for comparing aplurality of treatment plans, in accordance with one or more embodimentsherein.

DETAILED DESCRIPTION

A better understanding of the features and advantages of the presentdisclosure will be obtained by reference to the following detaileddescription that sets forth illustrative embodiments, in which theprinciples of embodiments of the present disclosure are utilized, andthe accompanying drawings.

Although the detailed description contains many specifics, these shouldnot be construed as limiting the scope of the disclosure but merely asillustrating different examples and aspects of the present disclosure.It should be appreciated that the scope of the disclosure includes otherembodiments not discussed in detail above. Various other modifications,changes and variations which will be apparent to those skilled in theart may be made in the arrangement, operation and details of themethods, systems, and apparatus of the present disclosure providedherein without departing from the spirit and scope of the invention asdescribed herein.

As used herein the terms “dental appliance,” and “tooth receivingappliance” are treated synonymously. As used herein, a “dentalpositioning appliance” or an “orthodontic appliance” may be treatedsynonymously, and may include any dental appliance configured to changethe position of a patient's teeth in accordance with a plan, such as anorthodontic treatment plan. A “dental positioning appliance” or“orthodontic appliance,” as used herein, may include a set of dentalappliances configured to incrementally change the position of apatient's teeth over time. As noted herein, dental positioningappliances and/or orthodontic appliances may comprise polymericappliances configured to move a patient's teeth in accordance with anorthodontic treatment plan.

As used herein the term “and/or” may be used as a functional word toindicate that two words or expressions are to be taken together orindividually. For example, the phrase “A and/or B” encompasses A alone,B alone, and A and B together. Depending on context, the term “or” neednot exclude one of a plurality of words/expressions. As an example, thephrase “A or B” need not exclude A and B together.

As used herein the terms “torque” and “moment” are treated synonymously.

As used herein a “moment” may encompass a force acting on an object suchas a tooth at a distance from a center of resistance. The moment may becalculated with a vector cross product of a vector force applied to alocation corresponding to a displacement vector from the center ofresistance, for example. The moment may comprise a vector pointing in adirection. A moment opposing another moment may encompass one of themoment vectors oriented toward a first side of the object such as thetooth and the other moment vector oriented toward an opposite side ofthe object such as tooth, for example. Any discussion herein referringto application of forces on a patient's teeth is equally applicable toapplication of moments on the teeth, and vice-versa.

As used herein a “plurality of teeth” may encompass two or more teeth. Aplurality of teeth may, but need not, comprise adjacent teeth. In someembodiments, one or more posterior teeth comprises one or more of amolar, a premolar or a canine, and one or more anterior teeth comprisingone or more of a central incisor, a lateral incisor, a cuspid, a firstbicuspid or a second bicuspid.

The embodiments disclosed herein may be well suited for moving one ormore teeth of the first group of one or more teeth or moving one or moreof the second group of one or more teeth, and combinations thereof.

The embodiments disclosed herein may be well suited for combination withone or more commercially available tooth moving components such asattachments and polymeric shell appliances. In some embodiments, theappliance and one or more attachments are configured to move one or moreteeth along a tooth movement vector comprising six degrees of freedom,in which three degrees of freedom are rotational and three degrees offreedom are translation.

The present disclosure provides orthodontic appliances and relatedsystems, methods, and devices. Repositioning of teeth may beaccomplished with the use of a series of removable elastic positioningappliances such as the Invisalign® system available from AlignTechnology, Inc., the assignee of the present disclosure. Suchappliances may have a thin shell of elastic material that generallyconforms to a patient's teeth but is slightly out of alignment with aninitial or immediately prior tooth configuration. Placement of theappliance over the teeth applies controlled forces in specific locationsto gradually move the teeth into the new configuration. Repetition ofthis process with successive appliances comprising new configurationseventually moves the teeth through a series of intermediateconfigurations or alignment patterns to a final desired configuration.Repositioning of teeth may be accomplished through other series ofremovable orthodontic and/or dental appliances, including polymericshell appliances.

Although reference is made to an appliance comprising a polymeric shellappliance, the embodiments disclosed herein are well suited for use withmany appliances that receive teeth, for example appliances without oneor more of polymers or shells. The appliance can be fabricated with oneor more of many materials such as metal, glass, reinforced fibers,carbon fiber, composites, reinforced composites, aluminum, biologicalmaterials, and combinations thereof for example. The appliance can beshaped in many ways, such as with thermoforming or direct fabrication asdescribed herein, for example. Alternatively or in combination, theappliance can be fabricated with machining such as an appliancefabricated from a block of material with computer numeric controlmachining. Additionally, though reference is made herein to orthodonticappliances, at least some of the techniques described herein may applyto restorative and/or other dental appliances, including withoutlimitation crowns, veneers, teeth-whitening appliances, teeth-protectiveappliances, etc.

Turning now to the drawings, in which like numbers designate likeelements in the various figures, FIG. 1A illustrates an exemplary toothrepositioning appliance or aligner 100 that can be worn by a patient inorder to achieve an incremental repositioning of individual teeth 102 inthe jaw. The appliance can include a shell (e.g., a continuous polymericshell or a segmented shell) having teeth-receiving cavities that receiveand resiliently reposition the teeth. An appliance or portion(s) thereofmay be indirectly fabricated using a physical model of teeth. Forexample, an appliance (e.g., polymeric appliance) can be formed using aphysical model of teeth and a sheet of suitable layers of polymericmaterial. The physical model (e.g., physical mold) of teeth can beformed through a variety of techniques, including 3D printing. Theappliance can be formed by thermoforming the appliance over the physicalmodel. In some embodiments, a physical appliance is directly fabricated,e.g., using additive manufacturing techniques, from a digital model ofan appliance. In some embodiments, the physical appliance may be createdthrough a variety of direct formation techniques, such as 3D printing.An appliance can fit over all teeth present in an upper or lower jaw, orless than all of the teeth. The appliance can be designed specificallyto accommodate the teeth of the patient (e.g., the topography of thetooth-receiving cavities matches the topography of the patient's teeth),and may be fabricated based on positive or negative models of thepatient's teeth generated by impression, scanning, and the like.Alternatively, the appliance can be a generic appliance configured toreceive the teeth, but not necessarily shaped to match the topography ofthe patient's teeth. In some cases, only certain teeth received by anappliance will be repositioned by the appliance while other teeth canprovide a base or anchor region for holding the appliance in place as itapplies force against the tooth or teeth targeted for repositioning. Insome cases, some or most, and even all, of the teeth will berepositioned at some point during treatment. Teeth that are moved canalso serve as a base or anchor for holding the appliance as it is wornby the patient. In some embodiments, no wires or other means will beprovided for holding an appliance in place over the teeth. In somecases, however, it may be desirable or necessary to provide individualattachments or other anchoring elements 104 on teeth 102 withcorresponding receptacles or apertures 106 in the appliance 100 so thatthe appliance can apply a selected force on the tooth. Exemplaryappliances, including those utilized in the Invisalign® System, aredescribed in numerous patents and patent applications assigned to AlignTechnology, Inc. including, for example, in U.S. Pat. Nos. 6,450,807,and 5,975,893, as well as on the company's website, which is accessibleon the World Wide Web (see, e.g., the url “invisalign.com”). Examples oftooth-mounted attachments suitable for use with orthodontic appliancesare also described in patents and patent applications assigned to AlignTechnology, Inc., including, for example, U.S. Pat. Nos. 6,309,215 and6,830,450.

Optionally, in cases involving more complex movements or treatmentplans, it may be beneficial to utilize auxiliary components (e.g.,features, accessories, structures, devices, components, and the like) inconjunction with an orthodontic appliance. Examples of such accessoriesinclude but are not limited to elastics, wires, springs, bars, archexpanders, palatal expanders, twin blocks, occlusal blocks, bite ramps,mandibular advancement splints, bite plates, pontics, hooks, brackets,headgear tubes, springs, bumper tubes, palatal bars, frameworks,pin-and-tube apparatuses, buccal shields, buccinator bows, wire shields,lingual flanges and pads, lip pads or bumpers, protrusions, divots, andthe like. In some embodiments, the appliances, systems and methodsdescribed herein include improved orthodontic appliances with integrallyformed features that are shaped to couple to such auxiliary components,or that replace such auxiliary components.

FIG. 1B illustrates a tooth repositioning system 110 including aplurality of appliances 112, 114, 116. Any of the appliances describedherein can be designed and/or provided as part of a set of a pluralityof appliances used in a tooth repositioning system. Each appliance maybe configured so a tooth-receiving cavity has a geometry correspondingto an intermediate or final tooth arrangement intended for theappliance. The patient's teeth can be progressively repositioned from aninitial tooth arrangement towards a target tooth arrangement by placinga series of incremental position adjustment appliances over thepatient's teeth. For example, the tooth repositioning system 110 caninclude a first appliance 112 corresponding to an initial tootharrangement, one or more intermediate appliances 114 corresponding toone or more intermediate arrangements, and a final appliance 116corresponding to a target arrangement. A target tooth arrangement can bea planned final tooth arrangement selected for the patient's teeth atthe end of all planned orthodontic treatment. Alternatively, a targetarrangement can be one of some intermediate arrangements for thepatient's teeth during the course of orthodontic treatment, which mayinclude various different treatment scenarios, including, but notlimited to, instances where surgery is recommended, where interproximalreduction (IPR) is appropriate, where a progress check is scheduled,where anchor placement is best, where palatal expansion is desirable,where restorative dentistry is involved (e.g., inlays, onlays, crowns,bridges, implants, veneers, and the like), etc. As such, it isunderstood that a target tooth arrangement can be any planned resultingarrangement for the patient's teeth that follows one or more incrementalrepositioning stages. Likewise, an initial tooth arrangement can be anyinitial arrangement for the patient's teeth that is followed by one ormore incremental repositioning stages.

FIG. 1C illustrates a method 150 of orthodontic treatment using aplurality of appliances, in accordance with embodiments. The method 150can be practiced using any of the appliances or appliance sets describedherein. In block 160, a first orthodontic appliance is applied to apatient's teeth in order to reposition the teeth from a first tootharrangement to a second tooth arrangement. In block 170, a secondorthodontic appliance is applied to the patient's teeth in order toreposition the teeth from the second tooth arrangement to a third tootharrangement. The method 150 can be repeated as necessary using anysuitable number and combination of sequential appliances in order toincrementally reposition the patient's teeth from an initial arrangementto a target arrangement. The appliances can be generated all at the samestage or in sets or batches (at the beginning of a stage of thetreatment, at an intermediate stage of treatment, etc.), or theappliances can be fabricated one at a time, and the patient can weareach appliance until the pressure of each appliance on the teeth can nolonger be felt or until the maximum amount of expressed tooth movementfor that given stage has been achieved. A plurality of differentappliances (e.g., a set) can be designed and even fabricated prior tothe patient wearing any appliance of the plurality. After wearing anappliance for an appropriate period of time, the patient can replace thecurrent appliance with the next appliance in the series until no moreappliances remain. The appliances are generally not affixed to the teethand the patient may place and replace the appliances at any time duringthe procedure (e.g., patient-removable appliances). The final applianceor several appliances in the series may have a geometry or geometriesselected to overcorrect the tooth arrangement. For instance, one or moreappliances may have a geometry that would (if fully achieved) moveindividual teeth beyond the tooth arrangement that has been selected asthe “final.” Such over-correction may be desirable in order to offsetpotential relapse after the repositioning method has been terminated(e.g., permit movement of individual teeth back toward theirpre-corrected positions). Over-correction may also be beneficial tospeed the rate of correction (e.g., an appliance with a geometry that ispositioned beyond a desired intermediate or final position may shift theindividual teeth toward the position at a greater rate). In such cases,the use of an appliance can be terminated before the teeth reach thepositions defined by the appliance. Furthermore, over-correction may bedeliberately applied in order to compensate for any inaccuracies orlimitations of the appliance.

The various embodiments of the orthodontic appliances presented hereincan be fabricated in a wide variety of ways. In some embodiments, theorthodontic appliances herein (or portions thereof) can be producedusing direct fabrication, such as additive manufacturing techniques(also referred to herein as “3D printing) or subtractive manufacturingtechniques (e.g., milling). In some embodiments, direct fabricationinvolves forming an object (e.g., an orthodontic appliance or a portionthereof) without using a physical template (e.g., mold, mask etc.) todefine the object geometry. Additive manufacturing techniques can becategorized as follows: (1) vat photopolymerization (e.g.,stereolithography), in which an object is constructed layer by layerfrom a vat of liquid photopolymer resin; (2) material jetting, in whichmaterial is jetted onto a build platform using either a continuous ordrop on demand (DOD) approach; (3) binder jetting, in which alternatinglayers of a build material (e.g., a powder-based material) and a bindingmaterial (e.g., a liquid binder) are deposited by a print head; (4)fused deposition modeling (FDM), in which material is drawn though anozzle, heated, and deposited layer by layer; (5) powder bed fusion,including but not limited to direct metal laser sintering (DMLS),electron beam melting (EBM), selective heat sintering (SHS), selectivelaser melting (SLM), and selective laser sintering (SLS); (6) sheetlamination, including but not limited to laminated object manufacturing(LOM) and ultrasonic additive manufacturing (UAM); and (7) directedenergy deposition, including but not limited to laser engineering netshaping, directed light fabrication, direct metal deposition, and 3Dlaser cladding. For example, stereolithography can be used to directlyfabricate one or more of the appliances herein. In some embodiments,stereolithography involves selective polymerization of a photosensitiveresin (e.g., a photopolymer) according to a desired cross-sectionalshape using light (e.g., ultraviolet light). The object geometry can bebuilt up in a layer-by-layer fashion by sequentially polymerizing aplurality of object cross-sections. As another example, the appliancesherein can be directly fabricated using selective laser sintering. Insome embodiments, selective laser sintering involves using a laser beamto selectively melt and fuse a layer of powdered material according to adesired cross-sectional shape in order to build up the object geometry.As yet another example, the appliances herein can be directly fabricatedby fused deposition modeling. In some embodiments, fused depositionmodeling involves melting and selectively depositing a thin filament ofthermoplastic polymer in a layer-by-layer manner in order to form anobject. In yet another example, material jetting can be used to directlyfabricate the appliances herein. In some embodiments, material jettinginvolves jetting or extruding one or more materials onto a build surfacein order to form successive layers of the object geometry.

In some embodiments, the direct fabrication methods provided hereinbuild up the object geometry in a layer-by-layer fashion, withsuccessive layers being formed in discrete build steps. Alternatively orin combination, direct fabrication methods that allow for continuousbuild-up of an object's geometry can be used, referred to herein as“continuous direct fabrication.” Various types of continuous directfabrication methods can be used. As an example, in some embodiments, theappliances herein are fabricated using “continuous liquid interphaseprinting,” in which an object is continuously built up from a reservoirof photopolymerizable resin by forming a gradient of partially curedresin between the building surface of the object and apolymerization-inhibited “dead zone.” In some embodiments, asemi-permeable membrane is used to control transport of aphotopolymerization inhibitor (e.g., oxygen) into the dead zone in orderto form the polymerization gradient. Continuous liquid interphaseprinting can achieve fabrication speeds about 25 times to about 100times faster than other direct fabrication methods, and speeds about1000 times faster can be achieved with the incorporation of coolingsystems. Continuous liquid interphase printing is described in U.S.Patent Publication Nos. 2015/0097315, 2015/0097316, and 2015/0102532,(corresponding to U.S. Pat. Nos. 9,205,601, 9,216,546, and 9,211,678)the disclosures of each of which are incorporated herein by reference intheir entirety.

As another example, a continuous direct fabrication method can achievecontinuous build-up of an object geometry by continuous movement of thebuild platform (e.g., along the vertical or Z-direction) during theirradiation phase, such that the hardening depth of the irradiatedphotopolymer is controlled by the movement speed. Accordingly,continuous polymerization of material on the build surface can beachieved. Such methods are described in U.S. Pat. No. 7,892,474, thedisclosure of which is incorporated herein by reference in its entirety.

In another example, a continuous direct fabrication method can involveextruding a composite material composed of a curable liquid materialsurrounding a solid strand. The composite material can be extruded alonga continuous 3D path in order to form the object. Such methods aredescribed in U.S. Patent Publication No. 2014/0061974, corresponding toU.S. Pat. No. 9,511,543, the disclosures of which are incorporatedherein by reference in its entirety.

In yet another example, a continuous direct fabrication method utilizesa “heliolithography” approach in which the liquid photopolymer is curedwith focused radiation while the build platform is continuously rotatedand raised. Accordingly, the object geometry can be continuously builtup along a spiral build path. Such methods are described in U.S. PatentPublication No. 2014/0265034, corresponding to U.S. Pat. No. 9,321,215,the disclosures of which are incorporated herein by reference in itsentirety.

The direct fabrication approaches provided herein are compatible with awide variety of materials, including but not limited to one or more ofthe following: polymer matrix reinforced with ceramic or metallicpolymers, a polyester, a co-polyester, a polycarbonate, a thermoplasticpolyurethane, a polypropylene, a polyethylene, a polypropylene andpolyethylene copolymer, an acrylic, a cyclic block copolymer, apolyetheretherketone, a polyamide, a polyethylene terephthalate, apolybutylene terephthalate, a polyetherimide, a polyethersulfone, apolytrimethylene terephthalate, a styrenic block copolymer (SBC), asilicone rubber, an elastomeric alloy, a thermoplastic elastomer (TPE),a thermoplastic vulcanizate (TPV) elastomer, a polyurethane elastomer, ablock copolymer elastomer, a polyolefin blend elastomer, a thermoplasticco-polyester elastomer, a thermoplastic polyamide elastomer, orcombinations thereof. The materials used for direct fabrication can beprovided in an uncured form (e.g., as a liquid, resin, powder, etc.) andcan be cured (e.g., by photopolymerization, light curing, gas curing,laser curing, crosslinking, etc.) in order to form an orthodonticappliance or a portion thereof. The properties of the material beforecuring may differ from the properties of the material after curing. Oncecured, the materials herein can exhibit sufficient strength, stiffness,durability, biocompatibility, etc. for use in an orthodontic appliance.The post-curing properties of the materials used can be selectedaccording to the desired properties for the corresponding portions ofthe appliance.

In some embodiments, relatively rigid portions of the orthodonticappliance can be formed via direct fabrication using one or more of thefollowing materials: a polyester, a co-polyester, a polycarbonate, athermoplastic polyurethane, a polypropylene, a polyethylene, apolypropylene and polyethylene copolymer, an acrylic, a cyclic blockcopolymer, a polyetheretherketone, a polyamide, a polyethyleneterephthalate, a polybutylene terephthalate, a polyetherimide, apolyethersulfone, and/or a polytrimethylene terephthalate.

In some embodiments, relatively elastic portions of the orthodonticappliance can be formed via direct fabrication using one or more of thefollowing materials: a styrenic block copolymer (SBC), a siliconerubber, an elastomeric alloy, a thermoplastic elastomer (TPE), athermoplastic vulcanizate (TPV) elastomer, a polyurethane elastomer, ablock copolymer elastomer, a polyolefin blend elastomer, a thermoplasticco-polyester elastomer, and/or a thermoplastic polyamide elastomer.

Optionally, the direct fabrication methods described herein allow forfabrication of an appliance including multiple materials, referred toherein as “multi-material direct fabrication.” In some embodiments, amulti-material direct fabrication method involves concurrently formingan object from multiple materials in a single manufacturing step usingthe same fabrication machine and method. For instance, a multi-tipextrusion apparatus can be used to selectively dispense multiple typesof materials (e.g., resins, liquids, solids, or combinations thereof)from distinct material supply sources in order to fabricate an objectfrom a plurality of different materials. Such methods are described inU.S. Pat. No. 6,749,414, the disclosure of which is incorporated hereinby reference in its entirety. Alternatively or in combination, amulti-material direct fabrication method can involve forming an objectfrom multiple materials in a plurality of sequential manufacturingsteps. For instance, a first portion of the object can be formed from afirst material in accordance with any of the direct fabrication methodsherein, then a second portion of the object can be formed from a secondmaterial in accordance with methods herein, and so on, until theentirety of the object has been formed. The relative arrangement of thefirst and second portions can be varied as desired, e.g., the firstportion can be partially or wholly encapsulated by the second portion ofthe object. The sequential manufacturing steps can be performed usingthe same fabrication machine or different fabrication machines, and canbe performed using the same fabrication method or different fabricationmethods. For example, a sequential multi-manufacturing procedure caninvolve forming a first portion of the object using stereolithographyand a second portion of the object using fused deposition modeling.

Direct fabrication can provide various advantages compared to othermanufacturing approaches. For instance, in contrast to indirectfabrication, direct fabrication permits production of an orthodonticappliance without utilizing any molds or templates for shaping theappliance, thus reducing the number of manufacturing steps involved andimproving the resolution and accuracy of the final appliance geometry.Additionally, direct fabrication permits precise control over the 3Dgeometry of the appliance, such as the appliance thickness. Complexstructures and/or auxiliary components can be formed integrally as asingle piece with the appliance shell in a single manufacturing step,rather than being added to the shell in a separate manufacturing step.In some embodiments, direct fabrication is used to produce appliancegeometries that would be difficult to create using alternativemanufacturing techniques, such as appliances with very small or finefeatures, complex geometric shapes, undercuts, interproximal structures,shells with variable thicknesses, and/or internal structures (e.g., forimproving strength with reduced weight and material usage). For example,in some embodiments, the direct fabrication approaches herein permitfabrication of an orthodontic appliance with feature sizes of less thanor equal to about 5 μm, or within a range from about 5 μm to about 50μm, or within a range from about 20 μm to about 50 μm.

In some embodiments, the direct fabrication methods described hereinimplement process controls for various machine parameters of a directfabrication system or device in order to ensure that the resultantappliances are fabricated with a high degree of precision. Suchprecision can be beneficial for ensuring accurate delivery of a desiredforce system to the teeth in order to effectively elicit toothmovements. Process controls can be implemented to account for processvariability arising from multiple sources, such as the materialproperties, machine parameters, environmental variables, and/orpost-processing parameters.

Material properties may vary depending on the properties of rawmaterials, purity of raw materials, and/or process variables duringmixing of the raw materials. In many embodiments, resins or othermaterials for direct fabrication should be manufactured with tightprocess control to ensure little variability in photo-characteristics,material properties (e.g., viscosity, surface tension), physicalproperties (e.g., modulus, strength, elongation) and/or thermalproperties (e.g., glass transition temperature, heat deflectiontemperature). Process control for a material manufacturing process canbe achieved with screening of raw materials for physical propertiesand/or control of temperature, humidity, and/or other process parametersduring the mixing process. By implementing process controls for thematerial manufacturing procedure, reduced variability of processparameters and more uniform material properties for each batch ofmaterial can be achieved. Residual variability in material propertiescan be compensated with process control on the machine, as discussedfurther herein.

Machine parameters can include curing parameters. For digital lightprocessing (DLP)-based curing systems, curing parameters can includepower, curing time, and/or grayscale of the full image. For laser-basedcuring systems, curing parameters can include power, speed, beam size,beam shape and/or power distribution of the beam. For printing systems,curing parameters can include material drop size, viscosity, and/orcuring power. These machine parameters can be monitored and adjusted ona regular basis (e.g., some parameters at every 1-x layers and someparameters after each build) as part of the process control on thefabrication machine. Process control can be achieved by including asensor on the machine that measures power and other beam parametersevery layer or every few seconds and automatically adjusts them with afeedback loop. For DLP machines, gray scale can be measured andcalibrated before, during, and/or at the end of each build, and/or atpredetermined time intervals (e.g., every n^(th) build, once per hour,once per day, once per week, etc.), depending on the stability of thesystem. In addition, material properties and/or photo-characteristicscan be provided to the fabrication machine, and a machine processcontrol module can use these parameters to adjust machine parameters(e.g., power, time, gray scale, etc.) to compensate for variability inmaterial properties. By implementing process controls for thefabrication machine, reduced variability in appliance accuracy andresidual stress can be achieved.

In many embodiments, environmental variables (e.g., temperature,humidity, Sunlight or exposure to other energy/curing source) aremaintained in a tight range to reduce variable in appliance thicknessand/or other properties. Optionally, machine parameters can be adjustedto compensate for environmental variables.

In many embodiments, post-processing of appliances includes cleaning,post-curing, and/or support removal processes. Relevant post-processingparameters can include purity of cleaning agent, cleaning pressureand/or temperature, cleaning time, post-curing energy and/or time,and/or consistency of support removal process. These parameters can bemeasured and adjusted as part of a process control scheme. In addition,appliance physical properties can be varied by modifying thepost-processing parameters. Adjusting post-processing machine parameterscan provide another way to compensate for variability in materialproperties and/or machine properties.

Although various embodiments herein are described with respect to directfabrication techniques, it shall be appreciated that other techniquescan also be used, such as indirect fabrication techniques. In someembodiments, the appliances herein (or portions thereof) can be producedusing indirect fabrication techniques, such as by thermoforming over apositive or negative mold. Indirect fabrication of an orthodonticappliance can involve one or more of the following steps: producing apositive or negative mold of the patient's dentition in a targetarrangement (e.g., by additive manufacturing, milling, etc.),thermoforming one or more sheets of material over the mold in order togenerate an appliance shell, forming one or more structures in the shell(e.g., by cutting, etching, etc.), and/or coupling one or morecomponents to the shell (e.g., by extrusion, additive manufacturing,spraying, thermoforming, adhesives, bonding, fasteners, etc.).Optionally, one or more auxiliary appliance components as describedherein (e.g., elastics, wires, springs, bars, arch expanders, palatalexpanders, twin blocks, occlusal blocks, bite ramps, mandibularadvancement splints, bite plates, pontics, hooks, brackets, headgeartubes, bumper tubes, palatal bars, frameworks, pin-and-tube apparatuses,buccal shields, buccinator bows, wire shields, lingual flanges and pads,lip pads or bumpers, protrusions, divots, etc.) are formed separatelyfrom and coupled to the appliance shell (e.g., via adhesives, bonding,fasteners, mounting features, etc.) after the shell has been fabricated.

In some embodiments, the orthodontic appliances herein can be fabricatedusing a combination of direct and indirect fabrication techniques, suchthat different portions of an appliance can be fabricated usingdifferent fabrication techniques and assembled in order to form thefinal appliance. For example, an appliance shell can be formed byindirect fabrication (e.g., thermoforming), and one or more structuresor components as described herein (e.g., auxiliary components, powerarms, etc.) can be added to the shell by direct fabrication (e.g.,printing onto the shell).

The configuration of the orthodontic appliances herein can be determinedaccording to a treatment plan for a patient, e.g., a treatment planinvolving successive administration of a plurality of appliances forincrementally repositioning teeth. Computer-based treatment planningand/or appliance manufacturing methods can be used in order tofacilitate the design and fabrication of appliances. For instance, oneor more of the appliance components described herein can be digitallydesigned and fabricated with the aid of computer-controlledmanufacturing devices (e.g., computer numerical control (CNC) milling,computer-controlled additive manufacturing such as 3D printing, etc.).The computer-based methods presented herein can improve the accuracy,flexibility, and convenience of appliance fabrication.

In some embodiments, computer-based 3D planning/design tools, such asTreat™ software from Align Technology, Inc., may be used to design andfabricate the orthodontic appliances described herein.

FIG. 2 illustrates a method 200 for designing an orthodontic applianceto be fabricated, in accordance with embodiments. The method 200 can beapplied to any embodiment of the orthodontic appliances describedherein. Some or all of the operations of the method 200 can be performedby any suitable data processing system or device, e.g., one or moreprocessors configured with suitable instructions.

In block 210, a movement path to move one or more teeth from an initialarrangement to a target arrangement is determined. The initialarrangement can be determined from a mold or a scan of the patient'steeth or mouth tissue, e.g., using wax bites, direct contact scanning,x-ray imaging, tomographic imaging, sonographic imaging, and othertechniques for obtaining information about the position and structure ofthe teeth, jaws, gums and other orthodontically relevant tissue. Fromthe obtained data, a digital data set can be derived that represents theinitial (e.g., pretreatment) arrangement of the patient's teeth andother tissues. Optionally, the initial digital data set is processed tosegment the tissue constituents from each other. For example, datastructures that digitally represent individual tooth crowns can beproduced. Advantageously, digital models of entire teeth can beproduced, including measured or extrapolated hidden surfaces and rootstructures, as well as surrounding bone and soft tissue.

The target arrangement of the teeth (e.g., a desired and intended endresult of orthodontic treatment) can be received from a clinician in theform of a prescription, can be calculated from basic orthodonticprinciples, and/or can be extrapolated computationally from a clinicalprescription. With a specification of the desired final positions of theteeth and a digital representation of the teeth themselves, the finalposition and surface geometry of each tooth can be specified to form acomplete model of the tooth arrangement at the desired end of treatment.

Having both an initial position and a target position for each tooth, amovement path can be defined for the motion of each tooth. In someembodiments, the movement paths are configured to move the teeth in thequickest fashion with the least amount of round-tripping to bring theteeth from their initial positions to their desired target positions.The tooth paths can optionally be segmented, and the segments can becalculated so that each tooth's motion within a segment stays withinthreshold limits of linear and rotational translation. In this way, theend points of each path segment can constitute a clinically viablerepositioning, and the aggregate of segment end points can constitute aclinically viable sequence of tooth positions, so that moving from onepoint to the next in the sequence does not result in a collision ofteeth.

In block 220, a force system to produce movement of the one or moreteeth along the movement path is determined. A force system can includeone or more forces and/or one or more torques. Different force systemscan result in different types of tooth movement, such as tipping,translation, rotation, extrusion, intrusion, root movement, etc.Biomechanical principles, modeling techniques, forcecalculation/measurement techniques, and the like, including knowledgeand approaches commonly used in orthodontia, may be used to determinethe appropriate force system to be applied to the tooth to accomplishthe tooth movement. In determining the force system to be applied,sources may be considered including literature, force systems determinedby experimentation or virtual modeling, computer-based modeling,clinical experience, minimization of unwanted forces, etc.

Determination of the force system can be performed in a variety of ways.For example, in some embodiments, the force system is determined on apatient-by-patient basis, e.g., using patient-specific data.Alternatively or in combination, the force system can be determinedbased on a generalized model of tooth movement (e.g., based onexperimentation, modeling, clinical data, etc.), such thatpatient-specific data is not necessarily used. In some embodiments,determination of a force system involves calculating specific forcevalues to be applied to one or more teeth to produce a particularmovement. Alternatively, determination of a force system can beperformed at a high level without calculating specific force values forthe teeth. For instance, block 220 can involve determining a particulartype of force to be applied (e.g., extrusive force, intrusive force,translational force, rotational force, tipping force, torqueing force,etc.) without calculating the specific magnitude and/or direction of theforce.

In block 230, an appliance geometry and/or material composition for anorthodontic appliance configured to produce the force system isdetermined. The appliance can be any embodiment of the appliancesdiscussed herein, such as an appliance having variable localizedproperties, integrally formed components, and/or power arms.

For example, in some embodiments, the appliance comprises aheterogeneous thickness, a heterogeneous stiffness, or a heterogeneousmaterial composition. In some embodiments, the appliance comprises twoor more of a heterogeneous thickness, a heterogeneous stiffness, or aheterogeneous material composition. In some embodiments, the appliancecomprises a heterogeneous thickness, a heterogeneous stiffness, and aheterogeneous material composition. The heterogeneous thickness,stiffness, and/or material composition can be configured to produce theforce system for moving the teeth, e.g., by preferentially applyingforces at certain locations on the teeth. For example, an appliance withheterogeneous thickness can include thicker portions that apply moreforce on the teeth than thinner portions. As another example, anappliance with heterogeneous stiffness can include stiffer portions thatapply more force on the teeth than more elastic portions. Variations instiffness can be achieved by varying the appliance thickness, materialcomposition, and/or degree of photopolymerization, as described herein.

In some embodiments, determining the appliance geometry and/or materialcomposition comprises determining the geometry and/or materialcomposition of one or more integrally formed components to be directlyfabricated with an appliance shell. The integrally formed component canbe any of the embodiments described herein. The geometry and/or materialcomposition of the integrally formed component(s) can be selected tofacilitate application of the force system onto the patient's teeth. Thematerial composition of the integrally formed component can be the sameas or different from the material composition of the shell.

In some embodiments, determining the appliance geometry comprisesdetermining the geometry for a variable gable bend.

The block 230 can involve analyzing the desired force system in order todetermine an appliance geometry and material composition that wouldproduce the force system. In some embodiments, the analysis involvesdetermining appliance properties (e.g., stiffness) at one or morelocations that would produce a desired force at the one or morelocations. The analysis can then involve determining an appliancegeometry and material composition at the one or more locations toachieve the specified properties. Determination of the appliancegeometry and material composition can be performed using a treatment orforce application simulation environment. A simulation environment caninclude, e.g., computer modeling systems, biomechanical systems orapparatus, and the like. Optionally, digital models of the applianceand/or teeth can be produced, such as finite element models. The finiteelement models can be created using computer program applicationsoftware available from a variety of vendors. For creating solidgeometry models, computer aided engineering (CAE) or computer aideddesign (CAD) programs can be used, such as the AutoCAD® softwareproducts available from Autodesk, Inc., of San Rafael, Calif. Forcreating finite element models and analyzing them, program products froma number of vendors can be used, including finite element analysispackages from ANSYS, Inc., of Canonsburg, Pa., and SIMULIA (Abaqus)software products from Dassault Systèmes of Waltham, Mass.

Optionally, one or more appliance geometries and material compositionscan be selected for testing or force modeling. As noted above, a desiredtooth movement, as well as a force system required or desired foreliciting the desired tooth movement, can be identified. Using thesimulation environment, a candidate appliance geometry and compositioncan be analyzed or modeled for determination of an actual force systemresulting from use of the candidate appliance. One or more modificationscan optionally be made to a candidate appliance, and force modeling canbe further analyzed as described, e.g., in order to iterativelydetermine an appliance design that produces the desired force system.

Optionally, block 230 can further involve determining the geometry ofone or more auxiliary components to be used in combination with theorthodontic appliance in order to exert the force system on the one ormore teeth. Such auxiliaries can include one or more of tooth-mountedattachments, elastics, wires, springs, bite blocks, arch expanders,wire-and-bracket appliances, shell appliances, headgear, or any otherorthodontic device or system that can be used in conjunction with theorthodontic appliances herein. The use of such auxiliary components maybe advantageous in situations where it is difficult for the appliancealone to produce the force system. Additionally, auxiliary componentscan be added to the orthodontic appliance in order to provide otherdesired functionalities besides producing the force system, such asmandibular advancement splints to treat sleep apnea, pontics to improveaesthetic appearance, and so on. In some embodiments, the auxiliarycomponents are fabricated and provided separately from the orthodonticappliance. Alternatively, the geometry of the orthodontic appliance canbe modified to include one or more auxiliary components as integrallyformed components.

In block 240, instructions for fabrication of the orthodontic appliancehaving the appliance geometry and material composition are generated.The instructions can be configured to control a fabrication system ordevice in order to produce the orthodontic appliance with the specifiedappliance geometry and material composition. In some embodiments, theinstructions are configured for manufacturing the orthodontic applianceusing direct fabrication (e.g., stereolithography, selective lasersintering, fused deposition modeling, 3D printing, continuous directfabrication, multi-material direct fabrication, etc.). Optionally, theinstructions can be configured to cause a fabrication machine todirectly fabricate the orthodontic appliance with teeth receivingcavities having variable gable bends, as discussed above and herein. Inalternative embodiments, the instructions can be configured for indirectfabrication of the appliance, e.g., by thermoforming.

Although the above blocks show a method 200 of designing an orthodonticappliance in accordance with some embodiments, a person of ordinaryskill in the art will recognize some variations based on the teachingdescribed herein. Some of the blocks may comprise sub-blocks. Some ofthe blocks may be repeated as often as desired. One or more blocks ofthe method 200 may be performed with any suitable fabrication system ordevice, such as the embodiments described herein. Some of the blocks maybe optional, and the order of the blocks can be varied as desired. Forinstance, in some embodiments, block 220 is optional, such that block230 involves determining the appliance geometry and/or materialcomposition based directly on the tooth movement path rather than basedon the force system.

FIG. 3 illustrates a method 300 for digitally planning an orthodontictreatment and/or design or fabrication of an appliance, in accordancewith embodiments. The method 300 can be applied to any of the treatmentprocedures described herein and can be performed by any suitable dataprocessing system.

In block 310, a digital representation of a patient's teeth is received.The digital representation can include surface topography data for thepatient's intraoral cavity (including teeth, gingival tissues, etc.).The surface topography data can be generated by directly scanning theintraoral cavity, a physical model (positive or negative) of theintraoral cavity, or an impression of the intraoral cavity, using asuitable scanning device (e.g., a handheld scanner, desktop scanner,etc.).

In block 320, one or more treatment stages are generated based on thedigital representation of the teeth. The treatment stages can beincremental repositioning stages of an orthodontic treatment proceduredesigned to move one or more of the patient's teeth from an initialtooth arrangement to a target arrangement. For example, the treatmentstages can be generated by determining the initial tooth arrangementindicated by the digital representation, determining a target tootharrangement, and determining movement paths of one or more teeth in theinitial arrangement necessary to achieve the target tooth arrangement.The movement path can be optimized based on minimizing the totaldistance moved, preventing collisions between teeth, avoiding toothmovements that are more difficult to achieve, or any other suitablecriteria.

In block 330, at least one orthodontic appliance is fabricated based onthe generated treatment stages. For example, a set of appliances can befabricated, each shaped according to a tooth arrangement specified byone of the treatment stages, such that the appliances can besequentially worn by the patient to incrementally reposition the teethfrom the initial arrangement to the target arrangement. The applianceset may include one or more of the orthodontic appliances describedherein. The fabrication of the appliance may involve creating a digitalmodel of the appliance to be used as input to a computer-controlledfabrication system. The appliance can be formed using direct fabricationmethods, indirect fabrication methods, or combinations thereof, asdesired.

In some instances, staging of various arrangements or treatment stagesmay not be necessary for design and/or fabrication of an appliance. Asillustrated by the dashed line in FIG. 3 , design and/or fabrication ofan orthodontic appliance, and perhaps a particular orthodontictreatment, may include use of a representation of the patient's teeth(e.g., receive a digital representation of the patient's teeth 310),followed by design and/or fabrication of an orthodontic appliance basedon a representation of the patient's teeth in the arrangementrepresented by the received representation.

Optionally, some or all of the blocks of the method 300 are performedlocally at the site where the patient is being treated and during asingle patient visit, referred to herein as “chair side manufacturing.”Chair side manufacturing can involve, for example, scanning thepatient's teeth, automatically generating a treatment plan withtreatment stages, and immediately fabricating one or more orthodonticappliance(s) to treat the patient using a chair side direct fabricationmachine, all at the treating professional's office during a singleappointment. In embodiments where a series of appliances are used totreat the patient, the first appliance may be produced chair side forimmediate delivery to the patient, with the remaining appliancesproduced separately (e.g., off site at a lab or central manufacturingfacility) and delivered at a later time (e.g., at a follow upappointment, mailed to the patient). Alternatively, the methods hereincan accommodate production and immediate delivery of the entire seriesof appliances on site during a single visit. Chair side manufacturingcan thus improve the convenience and speed of the treatment procedure byallowing the patient to immediately begin treatment at thepractitioner's office, rather than having to wait for fabrication anddelivery of the appliances at a later date. Additionally, chair sidemanufacturing can provide improved flexibility and efficiency oforthodontic treatment. For instance, in some embodiments, the patient isre-scanned at each appointment to determine the actual positions of theteeth, and the treatment plan is updated accordingly. Subsequently, newappliances can be immediately produced and delivered chair side toaccommodate any changes to or deviations from the treatment plan.

FIG. 4 is a simplified block diagram of a data processing system 400that may be used in executing methods and processes described herein.The data processing system 400 typically includes at least one processor402 that communicates with one or more peripheral devices via bussubsystem 404. These peripheral devices typically include a storagesubsystem 406 (memory subsystem 408 and file storage subsystem 414), aset of user interface input and output devices 418, and an interface tooutside networks 416. This interface is shown schematically as “NetworkInterface” block 416, and is coupled to corresponding interface devicesin other data processing systems via communication network interface424. Data processing system 400 can include, for example, one or morecomputers, such as a personal computer, workstation, mainframe, laptop,and the like.

The user interface input devices 418 are not limited to any particulardevice, and can typically include, for example, a keyboard, pointingdevice, mouse, scanner, interactive displays, touchpad, joysticks, etc.Similarly, various user interface output devices can be employed in asystem of the invention, and can include, for example, one or more of aprinter, display (e.g., visual, non-visual) system/subsystem,controller, projection device, audio output, and the like.

Storage subsystem 406 maintains the basic required programming,including computer readable media having instructions (e.g., operatinginstructions, etc.), and data constructs. The program modules discussedherein are typically stored in storage subsystem 406. Storage subsystem406 typically includes memory subsystem 408 and file storage subsystem414. Memory subsystem 408 typically includes a number of memories (e.g.,RAM 410, ROM 412, etc.) including computer readable memory for storageof fixed instructions, instructions and data during program execution,basic input/output system, etc. File storage subsystem 414 providespersistent (non-volatile) storage for program and data files, and caninclude one or more removable or fixed drives or media, hard disk,floppy disk, CD-ROM, DVD, optical drives, and the like. One or more ofthe storage systems, drives, etc., may be located at a remote location,such coupled via a server on a network or via the internet/World WideWeb. In this context, the term “bus subsystem” is used generically so asto include any mechanism for letting the various components andsubsystems communicate with each other as intended and can include avariety of suitable components/systems that would be known or recognizedas suitable for use therein. It will be recognized that variouscomponents of the system can be, but need not necessarily be at the samephysical location, but could be connected via various local-area orwide-area network media, transmission systems, etc.

Scanner 420 includes any means for obtaining a digital representation(e.g., images, surface topography data, etc.) of a patient's teeth(e.g., by scanning physical models of the teeth such as casts 421, byscanning impressions taken of the teeth, or by directly scanning theintraoral cavity), which can be obtained either from the patient or fromtreating professional, such as an orthodontist, and includes means ofproviding the digital representation to data processing system 400 forfurther processing. Scanner 420 may be located at a location remote withrespect to other components of the system and can communicate image dataand/or information to data processing system 400, for example, via anetwork interface 424. Fabrication system 422 fabricates appliances 423based on a treatment plan, including data set information received fromdata processing system 400. Fabrication machine 422 can, for example, belocated at a remote location and receive data set information from dataprocessing system 400 via network interface 424. The camera 425 mayinclude any image capture device configured to capture still images ormovies. The camera 425 may facilitate capturing various perspectives ofa patient's dentition. In some implementations, the camera 425 mayfacilitate capture of images at various focal lengths and distances fromthe patient.

The data processing aspects of the methods described herein can beimplemented in digital electronic circuitry, or in computer hardware,firmware, software, or suitable combinations thereof. Data processingapparatus can be implemented in a computer program product tangiblyembodied in a machine-readable storage device for execution by aprogrammable processor. Data processing blocks can be performed by aprogrammable processor executing program instructions to performfunctions by operating on input data and generating output. The dataprocessing aspects can be implemented in one or more computer programsthat are executable on a programmable system, the system including oneor more programmable processors operably coupled to a data storagesystem. Generally, a processor will receive instructions and data from aread-only memory and/or a random access memory. Storage devices suitablefor tangibly embodying computer program instructions and data includeall forms of nonvolatile memory, such as: semiconductor memory devices,such as EPROM, EEPROM, and flash memory devices; magnetic disks such asinternal hard disks and removable disks; magneto-optical disks; andCD-ROM disks.

Forming a Photo-Realistic Composite Image of a Patient's Dentition

In some embodiments, one or more modules of the data processing system400 may be configured to form a highly accurate composite image of apatient's dentition. As noted further herein, the processor(s) 402 mayexecute computer-implemented instructions stored on the storagesubsystem 406 to gather a 3D model of a patient's dentition. The 3Dmodel may include 3D virtual representations of a patient's dentitionfor display on a display device. In some implementations, the 3D modelmay facilitate modifications of parameters of teeth (locations, sizes,shapes, etc.) based on application of an orthodontic treatment plan. Forinstance, the 3D model may facilitate visualization of how a patient'steeth move and/or are represented after one or more orthodontictreatment plans have been applied to the patient's teeth. Theprocessor(s) 402 may execute computer-implemented instructions stored onthe storage subsystem 406 to gather an image of the patient that, insome embodiments, includes an image (e.g., a 2D image) of the patient'sface and dentition. The image may have been captured with a camera,uploaded with a phone or computer, and/or uploaded over a network (e.g.,the Internet).

In various implementations, the user interface and output device 418 mayreceive from a technician (e.g., a treatment professional) a first setof reference points corresponding to specific locations of the patient'sdentition as represented on the 3D model of the patient's dentition. Thefirst set of reference points may correspond to annotations, markups,drawn points, etc. placed on relevant anatomical points of the 3D model.The user interface and output device 148 may further receive from thetechnician a second set of reference points on a portion of the 2D imageof the patient. In some implementations, the second set of referencepoints may correspond to annotations, markups, drawn points, etc. placedon parts of the 2D image that correspond to the relevant anatomicalpoints identified by the first set of reference points on the 3D model.

In some implementations, the processor(s) 402 may execute programinstructions stored on the storage subsystem(s) 408 that combine the 3Dmodel of the patient's dentition with the image of the patient,particularly the dentition portion of the image of the patient. Such acombination may include identification of sizes, shapes, and/orperspectives of the various teeth modeled in the 3D model and/orrepresented in the image(s) of the patient's dentition. Combination mayfurther include adjusting and/or scaling the models and/orrepresentations of teeth so that specific parts of the 3D model have thesame size/shape/perspective, etc. as similar parts of 2D images.

In some implementations, the processor(s) 402 may execute programinstructions stored on the storage subsystem(s) 408 that align the firstset of reference points on the 3D model with the second set of referencepoints on the image of the patient, particularly portions of the imageof the patient corresponding to the patient's dentition. Alignment mayinvolve adjusting the 3D model to the image so that the 3D model moreaccurately models the patient's dentition. In some implementations, thealignment involves mathematically correlating points on the 3D modelwith points on the image. As an example, the alignment may involve astatistical technique to ensure distances between the representations ofthe first set of reference points on the 3D model and therepresentations of the second set of reference points on the image areminimized and/or otherwise optimized. In various implementations, thealignment involves creating a projection plane in a 3D space using theimage. The alignment, in such implementations, may further includeminimizing the sum of squares of distances between corresponding pointsin the projection plane and the 3D model of the patient's dentition.Within the 3D model, the processor(s) 402 may execute programinstructions stored on the storage subsystem(s) 408 to rotate and/orshift the 3D model of either jaw (mandible or maxilla) of a patient whenthe alignment is performed. Additionally, the processor(s) 402 mayexecute program instructions stored on the storage subsystem(s) 408 tochange focal lengths, zooms, and/or other aspects of perspectives in the3D model to facilitate visualization of a patient's dentition.

In some implementations, the processor(s) 402 may execute programinstructions stored on the storage subsystem(s) 408 to display amodified 2D representation of a patient's dentition. In someimplementations, the modified 2D representation may include estimatedresults of an orthodontic treatment plan. Rather than providing acomputerized or otherwise non-human depiction of the application of theorthodontic treatment plan, the modified 2D representation may show ahighly photo-realistic, emotion-evocative, humanistic visualization ofhow application of an orthodontic treatment plan will appear on thepatient. Such a modified rendering may solve technical problems relatedto computerized visualization of treatment planning by use of automatedimage matching processes.

FIG. 5A depicts a method 500 a of building a composite image, inaccordance with one or more embodiments herein. The method 500 a may beused with or in the processes and/or systems described above withreference to FIGS. 1-4 . A composite image, such as the composite image600 (shown in FIG. 6 ), can be helpful in the treatment planning processand treatment of a patient. By integrating a 3D bite model into a 2Dimage of a patient, the dental professional can evaluate the finalposition of the teeth while taking into account facial features. A 3Dbite model may be a 3D model of teeth, such as a patient's teeth. Inaddition, the treatment plan and proposed final or target position ofthe patient's teeth may be determined with the aid of the facial imageof the patient, thereby creating a treatment plan that positions theteeth based on a final orthodontic position that corrects malocclusionsand takes into account facial features, such as head shape, eyeposition, facial midline, and other facial features described herein.

To take into account the facial features of a patient, a composite imageof the patient is formed using a 2D image of the face of the patient anda 3D bite model generated from a scan of the patient's mouth or of apositive of negative model of a patient's teeth and gums. The 3D bitemodel of the patient with the teeth in an initial position is integratedinto the 2D image of the patient such that the teeth in the 3D bitemodel are placed in the same position at the teeth in the 2D image ofthe patient. At block 510 of method 500 a, a 3D bite model of thepatient's teeth is built. Such a model may be formed as described abovewith reference to FIGS. 1-4 . The 3D bite model includes a model of thepatient's teeth and gingiva in an initial position, such as beforetreatment, in a final or target position according to a treatment plan,or in an interim position that depicts the teeth and gums during thetreatment process. The system 400 may build the 3D bite model.

At block 520 a 2D image of the patient is received. In some embodiments,the 2D image includes the mouth of the patient and one or more of theface, head, neck, shoulders, torso, or the entire patient. The 2D imageof the patient may include an image of the patient with their mouth inone or more positions; for example, the patient's mouth may be a smilingposition, such as a social smiling position, a repose position withrelaxed muscles and lips slightly parted, or an anterior retracted openbite or closed bite positions.

In some embodiments, at block 520 an image of the patient is taken witha digital imaging device such as the camera 425. The image may becaptured with a lens at a predetermined focal length and at a distancefrom the patient. The image may be captured remotely and then receivedfor processing, for example, by the system 400 of FIG. 4 .

At block 530 reference points are selected or otherwise identified onthe 3D bite model and the 2D image of the patient. In some embodiments,the reference points may include the gingival apex of one or more teeth,such as the anterior teeth. In some embodiments, the reference pointsmay include the midpoint of the incisal edge of the teeth or the ends ofthe incisal edges of teeth. In some embodiments, the reference pointsmay be a cusp tip of one or more teeth, such as the cusp tips of thecanine teeth. In some embodiments, facial landmarks and contours,selected using neural networks, or otherwise, as descried herein, may beused as reference points. Examples of reference points are further shownin FIGS. 7, 8, 9, 14 and 15 and discussed further herein.

The references points of the 3D bite model correspond to referencepoints at the same location in the 2D image. For example, the referencepoints include the gingival apex of each of the six anterior teeth onthe 3D bite model and the reference points on the 2D image of thepatient can also include the gingival apex of each of the six anteriorteeth. Each of the reference points on a tooth or gingiva of the 3D bitemodel may correspond with a reference point on a tooth or gingiva of the2D image. For example, a reference point at the left incisal gingivalapex of the 3D bite model can correspond with the left incisal gingivalapex of the 2D image of the patient. Such two corresponding referencepoints may be considered a pair of corresponding reference points.

At block 540 the 3D bite model is combined with the 2D image of thepatient. The 3D bite model is manipulated and positioned such that theteeth of the 3D bite model are in the same position and the same size asthe teeth of the patient in the 2D image.

In some embodiments, the 3D bite model is positioned in the 2D image byaligning each of the reference points on the 3D bite model to each ofthe corresponding reference points on the 2D image of the patient. Insome embodiments, the reference points of the 3D bite model are alignedwith the corresponding reference points on the 2D image of the patientby minimizing the sum of the distance between each corresponding pair ofreference points. In some embodiments, the reference points of the 3Dbite model are aligned with the corresponding reference points on the 2Dimage of the patient by minimizing the sum of the squares of thedistance between each corresponding pair of reference points.

In some embodiments, combining the 3D bite model with the 2D image ofthe patient includes determining a mouth opening in the 2D image. Insome embodiments, the mouth opening is the shape of the inside edge ofthe patient's lips in the 2D image. In some embodiments, the portion ofthe 2D image within the mouth opening is removed or otherwise deletedfrom the 2D image of the patient. In some embodiments, the 3D bitemodel, or a 2D projection of the 3D bite model is placed or renderedbehind the 2D image such that the 3D bite model, or at least a portionof the 3D bite model, is visible through the mouth opening of the 2Dimage of the patient.

At block 550 the distance between one or more pairs of reference pointsor features is minimized. In some embodiments, the 3D bite model isrotated or translated about one or more of the three perpendicular axesof 3D space. In some embodiments, the size of the teeth relative to the2D image may be selected by matching the distance and focal length ofthe rendering of the 3D bite model with focal length and the distancebetween the imaging device and the patient used when capturing the 2Dimage of the patient.

At block 560 the composite image is formed. In some embodiments, thecomposite image is a composite 2D image that integrates the 2D image ofthe patient and a 2D rendering of the 3D bite model, or at least aportion of the 3D bite model, viewable through the mouth opening of the2D image. In some embodiments, the composite image is formed bydisplaying a 3D rendering of the 3D bite model, or at least a portion ofthe 3D bite model, behind and viewable through the mouth opening of the2D image of the patient on a surface or on a monitor. In variousimplementations, the composite image comprises a modified 2Drepresentation of a patient's dentition. In some implementations, themodified 2D representation may include estimated results of anorthodontic treatment plan. As noted herein, rather than providing acomputerized or otherwise non-human depiction of the application of theorthodontic treatment plan, the modified 2D representation may show ahighly photo-realistic, emotion-evocative, humanistic visualization ofhow application of an orthodontic treatment plan will appear on thepatient. Such a modified rendering may solve technical problems relatedto computerized visualization of treatment planning by use of automatedimage matching processes.

At block 570, the composite image is displayed. The composite image maybe displayed on a display device accessible to a patient locally,remotely, etc. In some implementations, the composite image is displayedremotely to the patient over a network connection (e.g., an Internetconnection) on a digital device of the patient. The composite image maydisplayed in substantial real-time (e.g., while the patient is in theoffice and getting images of their dentition taken) or after a visit. Insome implementations, the composite image is rendered into a format thatcan be displayed on a standalone application, a web page, a mobileapplication, and/or other portal for the patient.

FIG. 5B depicts a method 500 b of generating a virtual depiction of anorthodontic treatment of a patient, in accordance with one or moreembodiments herein. The method 500 b may be used with or in theprocesses and/or systems described above with reference to FIGS. 1-4 .The virtual depiction can take the form of a composite image, such asthe composite image 600 (shown in FIG. 6 ), which can be helpful in thetreatment planning process and treatment of a patient. At an operation522, a plurality of three-dimensional models of the patient's dentitionare generated. Each 3D model of the plurality may comprise a virtualrepresentation of the patient's dentition at a specific treatment stageof an orthodontic treatment plan. A 3D model of the plurality may begenerated based on a 3D scan of the patient's dentition, based on animpression of the patient's teeth, or based on a plurality of twodimensional images, for example. In some implementations, some or all ofthe plurality of 3D models are generated by gathering the 3D model asdescribed above with respect to FIG. 4 . In some implementations, the 3Dmodels are generated by accessing and loading into memory 3D models ofthe patient's teeth at different stages of a treatment plan that havebeen previously generated and stored in a computer-readable medium. The3D models can include 3D bite models, for example.

At an operation 524, an image of the patient is obtained. In someimplementations, the image includes at least a portion of the patient'sface and including at least a portion of the patient's dentition. Theimage can be obtained by taking a 2D or 3D image of the patient's face,for example, or retrieving a stored image of the patient from a computerreadable medium. In some implementations, obtaining the image includesgathering the image as discussed above with respect to FIG. 4 . At anoperation 526, first identifiers of a first set of reference pointsmodeled on a first three-dimensional model of the plurality ofthree-dimensional models of the patient's dentition are received. Thefirst set of reference points may correspond to a set of anatomicalpoints on the patient's dentition. In some implementations, the firstset of reference points are provided by a technician or other clinicalprofessional who has identified specific anatomical portions of the 3Drepresentation of interest to make photo-realistic and/or humanistic.The first set of reference points may be chosen to minimize “uncannyvalley” problems with machine representations of the patient'sdentition. In some embodiments, the reference points may include thegingival apex of one or more teeth, such as the anterior teeth. In someembodiments, the reference points may include the midpoint of theincisal edge of the teeth or the ends of the incisal edges of teeth. Insome embodiments, the reference points may be a cusp tip of one or moreteeth, such as the cusp tips of the canine teeth. In some embodiments,facial landmarks and contours, selected using neural networks, orotherwise, as descried herein, may be used as reference points. Examplesof reference points are further shown in FIGS. 7, 8, 9, 14 and 15 anddiscussed further herein.

At an operation 528, second identifiers of a second set of referencepoints modeled on the dentition of the image of the patient arereceived. The second set of reference points may correspond to the setof anatomical points on the patient's dentition. The second set ofreference points may be provided by the technician or other clinicalprofessional.

At an operation 532, the image of the patient's dentition may beprojected into a three-dimensional space to create a projected 3D modelof the image of the patient's dentition. In various implementations,points at 2D coordinates of the image may be mapped into a 3D space sothat representations of 2D features in the 3D space are identified. Insome implementations, the projection can be represented as a 2D plane ina 3D space. In some implementations, the projection can include depthinformation, producing a 3D model of the patient's dentition.

At an operation 534, the first set of reference points on the first 3Dmodel of the patient's dentition may be aligned with the second set ofreference points on the projected model of the image of the patient'sdentition. In various implementations, the distance between one or morepairs of reference points or features is minimized. In some embodiments,the first 3D model is rotated or translated about one or more of thethree perpendicular axes of 3D space. In some embodiments, the size ofthe teeth relative to the 2D image may be selected by matching thedistance and focal length of the rendering of the 3D model with focallength and the distance between the imaging device and the patient usedwhen capturing the 2D image of the patient.

At an operation 536, a plurality of modified images of the patient aregenerated based on the alignment performed in operation 534. A firstmodified image can be generated by forming a composite of the first 3Dmodel of the patient's dentition and the projected model of the image ofthe patient's dentition. The first 3D model can supply the locations andorientations of the patient's dentition, replacing the dentition in theprojected model. The projected model can supply the remaining portionsof the patient's face, such as the lips, nose, cheeks, etc. Thepatient's gums, where visible, can be derived entirely or at leastpartially from the first 3D model in some cases. The remainder of theplurality of modified images can be generated based on this firstmodified image. Each of the remainder of modified images can correspondto a different stage of the treatment plan, and the movements of thepatient's teeth can be included in the modified images by adjusting theteeth based on a comparison to another modified image of the pluralityof modified images. The comparison can take into account movement of theteeth in a respective stage of treatment, as reflected in the pluralityof 3D models, and the teeth in a given modified image can be moved basedon this reflected movement to generate a new modified image. Thisprocess can be repeated to generate the plurality of modified images,reflecting images of the patient including the patient's dentition ateach of a plurality of steps in the treatment plan. In some instances,the modified images can be 3D images. In some instances, the modifiedimages can be 2D images; for example, the modified images can beprojected into two dimensions from a plurality of modified 3D images togenerate a plurality of modified 2D images for display to the patient.

At an operation 538, instructions are provided to display at least onemodified image of the plurality of modified images of the patient. Themodified image may represent the aligned first and second sets ofreference points. In some implementations, the modified image is ahighly photo-realistic, humanistic image of the patient's dentition.Displaying a modified image can include displaying a 2D, photo-realisticimage based on a 3D version of the modified image. The modified imagemay include the estimated results of an intermediate or final stage ofthe orthodontic treatment plan. The modified image may accommodatevarious perspectives, zooms, angles, etc. of the patient's dentition asinformed through data modeled in the 3D model. The instructions can beused to display the at least one modified image to the patient. In somecases, the display can include an interface, such as a touchscreen or amouse, allowing a user to change the modified image being displayed to adifferent modified image of the plurality of modified images. In someimplementations, the plurality of modified images can be displayed as amovie, allowing the patient to see the changes to the patient'sdentition over the course of part or all of the treatment plan.Optionally, instructions can be sent to fabricate one or more appliancesbased on the treatment plan.

FIG. 5C depicts a method 500 c of generating virtual depictions of aplurality of alternative orthodontic treatments for a patient, accordingto one or more embodiments disclosed herein. The method 500 c may beused with or in the processes and/or systems described above withreference to FIGS. 1-4 . The virtual depictions can take the form of acomposite image, such as the composite image 600 (shown in FIG. 6 ),which can be helpful in the treatment planning process and treatment ofa patient. At an operation 542, a first plurality of three-dimensionalmodels of the patient's dentition is generated. Each 3D model of thefirst plurality may comprise a virtual representation of the patient'sdentition at a specific treatment stage of a first orthodontic treatmentplan. A 3D model of the first plurality of 3D models may be generatedbased on a 3D scan of the patient's dentition, based on an impression ofthe patient's teeth, or based on a plurality of two-dimensional images,for example. In some implementations, some or all of the plurality of 3Dmodels are generated by gathering the 3D model as described above withrespect to FIG. 4 . In some implementations, the 3D models are generatedby accessing and loading into memory 3D models of the patient's teeth atdifferent stages of a treatment plan that have been previously generatedand stored in a computer-readable medium. The 3D models can include 3Dbite models, for example.

At an operation 543, a second plurality of three-dimensional models ofthe patient's dentition is generated. Each 3D model of the secondplurality may comprise a virtual representation of the patient'sdentition at a specific treatment stage of a second orthodontictreatment plan. For example, the second treatment plan can be analternative treatment strategy to the first treatment plan. The secondtreatment plan might seek to provide the same treatment endpoint as thefirst plan using different means, or seek a different treatmentendpoint, for example. These differences can be reflected in differentconfigurations of teeth in the respective 3D models of the first andsecond pluralities. The second plurality of three-dimensional models caninclude a model in common with the first plurality of three-dimensionalmodels; for example, the two models can represent alternative treatmentplans beginning at a common starting point (e.g., the presentconfiguration of the patient's dentition). The second plurality of 3Dmodels can be generated using methods similar to those used to generatethe first plurality of 3D models, but using the second treatment plan todetermine tooth movements instead of the first treatment plan.

At an operation 544, an image of the patient is obtained. In someimplementations, the image includes at least a portion of the patient'sface and including at least a portion of the patient's dentition. Theimage can be obtained by taking a 2D or 3D image of the patient's face,for example, or retrieving a stored image of the patient from a computerreadable medium. In some implementations, obtaining the image includesgathering the image as discussed above with respect to FIG. 4 .

At an operation 546, first identifiers of a first set of referencepoints modeled on a first three-dimensional model of the patient'sdentition are received. The first three-dimensional model can be athree-dimensional model common between the first and second plurality ofthree-dimensional models. The common model can be an identical model ineach plurality. Alternatively, the two pluralities can contain a pair ofmodels with only minor differences, such that the pair of models are thesame for practical orthodontic purposes. In this latter case, eithermodel can be treated as the model common to both pluralities. The firstset of reference points may correspond to a set of anatomical points onthe patient's dentition. In some implementations, the first set ofreference points are provided by a technician or other clinicalprofessional who has identified specific anatomical portions of the 3Drepresentation of interest to make photo-realistic and/or humanistic.The first set of reference points may be chosen to minimize “uncannyvalley” problems with machine representations of the patient'sdentition. In some embodiments, the reference points may include thegingival apex of one or more teeth, such as the anterior teeth. In someembodiments, the reference points may include the midpoint of theincisal edge of the teeth or the ends of the incisal edges of teeth. Insome embodiments, the reference points may be a cusp tip of one or moreteeth, such as the cusp tips of the canine teeth. In some embodiments,facial landmarks and contours, selected using neural networks, orotherwise, as descried herein, may be used as reference points. Examplesof reference points are further shown in FIGS. 7, 8, 9, 14 and 15 anddiscussed further herein.

At an operation 548, second identifiers of a second set of referencepoints modeled on the dentition of the image of the patient arereceived. The second set of reference points may correspond to the setof anatomical points on the patient's dentition. The second set ofreference points may be provided by the technician or other clinicalprofessional.

At an operation 552, the image of the patient's dentition may beprojected into a three-dimensional space to create a projected 3D modelof the image of the patient's dentition. In various implementations,points at 2D coordinates of the image may be mapped into a 3D space sothat representations of 2D features in the 3D space are identified. Insome implementations, the projection can be represented as a 2D plane ina 3D space. In some implementations, the projection can include depthinformation, producing a 3D model of the patient's dentition.

At an operation 554, the first set of reference points on the first 3Dmodel of the patient's dentition may be aligned with the second set ofreference points on the projected model of the image of the patient'sdentition. In various implementations, the distance between one or morepairs of reference points or features is minimized. In some embodiments,the first 3D model is rotated or translated about one or more of thethree perpendicular axes of 3D space. In some embodiments, the size ofthe teeth relative to the 2D image may be selected by matching thedistance and focal length of the rendering of the 3D model with focallength and the distance between the imaging device and the patient usedwhen capturing the 2D image of the patient.

At an operation 556, a plurality of modified images of the patient aregenerated based on the alignment performed in operation 554. A firstmodified image can be generated by forming a composite of the first 3Dmodel of the patient's dentition and the projected model of the image ofthe patient's dentition. The first 3D model can supply the locations andorientations of the patient's dentition, replacing the dentition in theprojected model. The projected model can supply the remaining portionsof the patient's face, such as the lips, nose, cheeks, etc. Thepatient's gums, where visible, can be derived entirely or at leastpartially from the first 3D model in some cases. The remainder of theplurality of modified images can be generated based on this firstmodified image. Each of the remainder of modified images can correspondto a different stage of the treatment plan, and the movements of thepatient's teeth can be included in the modified images by adjusting theteeth based on a comparison to another modified image of the pluralityof modified images. The comparison can take into account movement of theteeth in a respective stage of treatment, as reflected in the pluralityof 3D models, and the teeth in a given modified image can be moved basedon this reflected movement to generate a new modified image. Thisprocess can be repeated to generate the plurality of modified images,reflecting images of the patient including the patient's dentition ateach of a plurality of steps in the treatment plan. In some instances,the modified images can be 3D images. In some instances, the modifiedimages can be 2D images; for example, the modified images can beprojected into two dimensions from a plurality of modified 3D images togenerate a plurality of modified 2D images for display to the patient.

At an operation 558, instructions are provided to display at least onemodified image of the plurality of modified images of the patient. Themodified image may represent the aligned first and second sets ofreference points. In some implementations, the modified image is ahighly photo-realistic, humanistic image of the patient's dentition.Displaying a modified image can include displaying a 2D, photo-realisticimage based on a 3D version of the modified image. The modified imagemay include the estimated results of an intermediate or final stage ofthe orthodontic treatment plan. The modified image may accommodatevarious perspectives, zooms, angles, etc. of the patient's dentition asinformed through data modeled in the 3D model. The instructions can beused to display the at least one modified image to the patient. In somecases, the display can include an interface, such as a touchscreen or amouse, allowing a user to change the modified image being displayed to adifferent modified image of the plurality of modified images. In someimplementations, the plurality of modified images can be displayed as amovie, allowing the patient to see the changes to the patient'sdentition over the course of part or all of the treatment plan.Optionally, instructions can be sent to fabricate one or more appliancesbased on the treatment plan.

FIG. 6 depicts a two-dimensional image 600 of a patient, in accordancewith one or more embodiments herein. The 2D image 600 may be received orcaptured as described above with respect to the method 500. The image600 of the patient includes a 2D image of the representation of thepatient's face that may be created by and received from an imagingdevice, such as a camera. In some implementations, the 2D image may havebeen captured with a camera, uploaded with a phone or computer, and/oruploaded over a network (e.g., the Internet). The 2D image 600 of thepatient's face includes a mouth opening 610 and the patient's teeth andgingiva. The patient's mouth may be defined by the inner edges of theupper lip 620 and the lower lip 630. In particular, the outer perimeterof the mouth opening 610 may be defined by the lower edge of the upperlips 620, or a portion of the lower edge of the upper lips 620, and theupper edge of the lower lips 630, or a portion of the upper edge of thelower lips 630. FIG. 13D shows a process of finding lip edges andcontours.

FIG. 7 depicts a 3D bite model 700 of a patient's teeth within themouth. The 3D bite model may be formed as described above with referenceto FIGS. 1-4 . As shown in FIG. 7 , the 3D bite model includes 3D modelsof the patient's upper and lower jaws, including the teeth 710 and gums720 of the upper and lower jaws of the patient. In some embodiments, the3D bite model is a model of the patient's teeth in a pre-treatment orinitial position. In some embodiments, the 3D bite model depicts theteeth in the same relative positions as the relative positions of theteeth in the 2D image of the patient. For example, the 2D image of thepatient and scan of the patient's teeth may have been created beforetreatment starts, such as, for example, on the same day or within a timeperiod such that the position of the teeth in the 3D scan from which the3D bite model is built substantially matches the position of the teethat the time the 2D image of the patient is captured.

FIG. 8 depicts the selection of reference points 810 a, 810 b, 820 a,820 b on the 2D image of a patent. The selection of the reference points810 a, 810 b, 820 a, 820 b may be selected according to the processdescribed above with respect to FIGS. 5A-C and block 530. In someembodiments, the reference points 810 a, 820 a may be the gingival apexof one or more teeth, such as the anterior teeth. In some embodiments,the reference point 810 b may be the midpoint of the incisal edge of theteeth. In some embodiments, the reference point 820 b may be a cusp tipof one or more teeth, such as the cusp tips of the canine teeth.

The references points of the 2D image may correspond to correspondingreference points at the same location on the 3D bite model. FIG. 9depicts the selection of reference points 910 a, 910 b, 920 a, 920 b onthe 3D bite model of the patient's teeth. The selection of the referencepoints 910 a, 910 b, 920 a, 920 b may be selected according to theprocess described above with respect to FIGS. 5A-C and block 530. Insome embodiments, the reference points 910 a, 920 a may be the gingivalapex of one or more teeth, such as the anterior teeth. In someembodiments, the reference point 910 b may be the midpoint of theincisal edge of the teeth. In some embodiments, the reference point 920b may be a cusp tip of one or more teeth, such as the cusp tips of thecanine teeth.

FIGS. 10 and 11 depict the integration of the 3D bite model 1040 intothe 2D image 600 of the patient as part of the process to create acomposite image 1000. The integration of the 3D bite model 1040 into the2D image 600 depicted in FIGS. 10 and 11 may proceed as described abovewith reference to FIGS. 5A-C and block 540. In some embodiments,combining the 3D bite model 1040 with the 2D image 600 of the patientincludes determining a mouth opening 1010 in the 2D image. In someembodiments, the mouth opening 1010 is the shape of the inside edge ofthe patient's lips 1020, 1030 in the 2D image. In some embodiments, theportion of the 2D image within the mouth opening 1010 is deleted orotherwise removed from the 2D image 600 of the patient. In someembodiments, the 3D bite model 1040, or a 2D projection of the 3D bitemodel 1040 is placed or rendered behind the 2D image 600 such that the3D bite model 1040, or at least a portion of the 3D bite model 1040, isvisible through the mouth opening of the 2D image of the patient.

As shown in FIG. 11 , the 3D bite model 1040 is manipulated andpositioned such that the teeth of the 3D bite model 1040 are in the sameposition and the same size as the teeth of the patient in the 2D image1000.

FIG. 11 depicts a close-up of a composite image 1000 of the 3D bitemodel and the 2D image 600 of the patient with overlaid contours 1110 a,1110 b and reference points 810 a, 810 b, 820 a, 820 b, 910 a, 910 b,920 a, 920 b. As depicted in FIG. 11 , two corresponding referencepoints may be considered a pair of corresponding reference points. Forexample, a reference point 910 a at the left incisal gingival apex ofthe 3D bite model can correspond with the reference points 810 a at theleft incisal gingival apex of the 2D image of the patient. As additionalexamples, reference points 810 b, 910 b are a corresponding pair ofreference points at respective incisal edge midlines, reference points820 a, 920 a are a corresponding pair of reference points at arespective gingival apex of the canine, and reference points 820 b, 920b are corresponded a pair of reference points at respective caninecusps.

The alignment of the 3D bite model with the 2D image may be performed asdescribed above with reference to FIGS. 5A-C and block 540 and elsewhereherein, for example as describe with reference to FIGS. 13A-F. The 3Dbite model 1040 is manipulated and positioned such that the teeth of the3D bite model 1040 are in the same position and the same size as theteeth of the patient in the 2D image 600. In some embodiments, the 3Dbite model 1040 is positioned in the 2D image 600 by aligning each ofthe reference points on the 3D bite model 1040 to each of thecorresponding reference points on the 2D image 600 of the patient. Insome embodiments, the reference points of the 3D bite model 1040 arealigned with the corresponding reference points on the 2D image 600 ofthe patient by minimizing the sum of the distance between eachcorresponding pair of reference points. In some embodiments, thereference points of the 3D bite model 1040 are aligned with thecorresponding reference points on the 2D image 600 of the patient byminimizing the sum of the squares of the distance between eachcorresponding pair of reference points.

In some embodiments, the 3D bite model 1040 and the 2D image 600 of thepatient may be aligned based on the alignment of tooth contours 1110 a,1110 b for the 3D bite model 1040 and the 2D image 600 of the patient.For example, the alignment may minimize the distance or square of thedistance between respective points that define the contours 1110 a, 1110b of the 3D bite model 1040 and the 2D image 600 of the patient.

In some embodiments, the 3D bite model may be aligned when the distancebetween all the reference points is less than a threshold, such as lessthan 0.05 mm, 0.1 mm, 0.2 mm, or 0.5 mm or wherein the sum of thesquares of the distances between the reference points is less than athreshold or minimized.

FIG. 12 depicts a composite image 1200 of the 3D bite model 1040 and the2D image of the patient with attachments 1210 placed on the patient'steeth. Attachments may be added to the 3D bite model during thetreatment planning process to aid in applying forces to the teeth.

FIG. 13 depicts a composite image 1300 of the 3D bite model 1040 withteeth in a final position and the 2D image of the patient. The 3D bitemodel 1040 and the 2D image of the patient may be aligned according toone or more embodiments described herein. Color and other featurematching may also be performed accordingly to one or more embodimentsdescribed herein.

FIG. 13A depicts a method 4600 for extracting mouth and teeth contoursfrom a patient's 2D image and aligning the image contours with contoursof a 3D model of the patients teeth.

At block 4610 the method 4600 determines the facial landmarks from afacial image. The facial image may be a two-dimensional facial image ofthe patient. The patient's facial landmarks within the image arecomputed using a machine learning algorithm, such as a convoluted neuralnetwork, to determine facial features such as the jawline, thecenterline of the face, the location of the eyes and eyebrows, thelocation of features of the nose, and also landmarks along the mouthopening and lips.

At block 4620 the shape and contours of the mouth opening are identifiedbased on the facial landmarks determined at block 4610. For example,each of the landmarks identified as being located on an inner lipcontour may be identified and contour lines may be drawn between each ofthe identified landmarks to determine the mouth opening. These faciallandmarks may be used to detect a mouth region and to conduct a courseplacing of the 3D model inside the mouth.

At block 4630 the contours within the mouth opening are extracted. Aconvolutional neural network may be used to extract the contours of thelips, gingiva, and teeth from within the mouth opening. In someembodiments holistic edge detection architecture of a convolutionalneural network is used to detect these and other contours describedherein.

At block 4640 the contours extracted from the facial image are matchedwith the contours of a three-dimensional model of the patient's teeth.In some embodiments the 3D model of the patient's teeth are in aninitial or pre-treatment position. For example the 3D model of thepatient's teeth may have been derived from a scan of the patient's teethtaken at or around the same time as the image of the patient's face.

FIGS. 13B and 13C depict some of the operations of the method 4600 forextracting mouth and teeth contours from a patient's two-dimensionalimage in more detail.

FIG. 13B depicts a facial image 4650 of a patient with facial landmarks4652 of a patient's face 4651 identified according to a machine learningalgorithm. The facial landmarks 4652 may include nasal centerlinelandmarks 4652A, facial outline landmarks 4652C, and inner lip landmarks4652D, among other landmarks such as landmarks that identify thepatient's ears, eyebrows, eyes, features of the nose, chin, and otherfacial features. These facial landmarks can then be used to identifyaspects of the patient's face. For example, the inner lip landmarks maybe used to identify the mouth opening, within which the teeth andgingiva are located.

FIG. 13C shows the identification of the patient's lips, teeth, andgingiva. At panel 4660, the initial determination of the patient's lips4664, teeth 4662, and gingiva 4666 contours are identified by a machinelearning algorithm, such as a convoluted neural network.

At panel 4670, the initial tooth contours 4674 are extracted from themouth area. In some embodiments, the lips, gingiva, and other facialcontours identified in the facial image are removed from the image,resulting in an extraction of the tooth contours 4674 shown in panel4670. The tooth contours 4674 shown in panel 4670 have a brightness orother scale applied to them. For example, in a grey scale image of thecontours, each pixel may be assigned a value between 0 and 255, whichmay indicate a confidence that the pixel is a contour or may indicatethe magnitude of the contour at that location.

At panel 4680, the tooth contours, and in particular, the pixels thatdenote the contours undergo binarization to change the pixels from ascale of, for example, 0 to 255, to a binary scale, for example, of 0 or1, creating binarized tooth contours 4682. In the binarization process,the value of each pixel is compared to a threshold value. If the valueof the pixel is greater than the threshold, then it may be assigned anew first value, such as a value of 1 and if the pixel is less than thethreshold, then it may be assigned a new second value, such as a valueof 0, for example.

At panel 4690, the binarized tooth contours 4682 from panel 4680 arethinned, whereby the contour's thickness is reduced to, for example, asingle pixel in width, forming a thinned tooth contour 4692. The widthof a contour for thinning may be measured as the shortest distance froma contour pixel adjacent to a non-contour pixel on a first side of acontour, to a contour pixel adjacent a non-contour pixel on a secondside of the contour. The single pixel representing the thinned contourat a particular location may be located at the midpoint of the widthbetween the pixel at the first side and the pixel at the second side.

After thinning the binarized tooth contours 4682, the thinned contour4692 may be a single width contour at a location that corresponds to amidpoint of the binarized contour 4682.

After forming the thinned contour 4682 of the teeth, the thinnedcontours 4682 may be combined with lip contours, described below, andthe facial image, as shown, for example, in FIG. 13E, discussed below.

FIG. 13D shows a process of identifying a patient's lip contours andmouth opening. At panel 4710 an image of a patient's face, and inparticular the mouth and the region near the mouth, is shown. The lipcontours and mouth opening are determined based on such an image.

The initial determination of the patient's lips may be based on faciallandmarks, such as the lip landmarks determined according to a machinelearning algorithm. For example, the facial landmarks may include innerlip landmarks 4652D shown in FIG. 13D.

In some embodiments, the 3D model of the patient's teeth is initiallycoarsely placed within the mouth region. The initial placement is basedon an alignment of the midline of the 3D model with a midline of thefacial image determined based on the midline facial landmarks, such asthe nasal midline landmarks 4652 a of FIG. 13B. The initial scale of the3D model may be determined by matching the scale of a distance betweentips of opposite teeth, such as, by matching the distance between theupper canines in the 3D model with the distance between the tips of theupper canines in the facial image or matching the size of one or moreteeth in the 3D model with the site of one or more corresponding teethin the facial image.

Panel 4720 shows the initial determination of the patient's lip contours4722, as identified by a machine learning algorithm, such as aconvoluted neural network. These initial lip contours 4722 are extractedfrom the mouth area. In some embodiments, the tooth, gingiva, and otherfacial contours identified in the facial image are removed from theimage, resulting in an extraction of the lip contours 4724 shown inpanel 4720. In some embodiments, other facial contours 4722 may bepresent at this stage in the process.

As with the tooth contours 4674 shown in panel 4670, the lip contours4722 have a brightness or other scale applied to them. For example, in agrey scale image of the contours, each pixel may be assigned a valuebetween 0 and 255, which may indicate a confidence that the pixel is acontour or may indicate the magnitude of the contour at that location.

At panel 4730, the lip contours, and in particular, the pixels thatdenote the contours undergo binarization to change the pixels from ascale of, for example, 0 to 255, to a binary scale of, for example, 0 or1, creating binarized lip contours 4732. The binarization process issimilar to that described above with respect to FIG. 13C.

At panel 4740, the binarized lip contours 4732 from panel 4730 arethinned, whereby the contour's thickness is reduced to, for example, asingle pixel in width, and forming a thinned lip contour 4742. Asdescribed above with respect to FIG. 13B, the width of a contour forthinning may be measured as the shortest distance from a contour pixeladjacent to a non-contour pixel on a first side of a contour, to acontour pixel adjacent a non-contour pixel on a second side of thecontour. The single pixel representing the thinned contour at aparticular location may be located at the midpoint of the width betweenthe pixel at the first side and the pixel at the second side.

After thinning the binarized lip contours 4732, the thinned contour 4742may be a single pixel width contour at a location that corresponds tothe midpoint of the binarized contour 4732. The thinned contour imagemay still include other facial contours 4744 that are not part of thelip contours 4742. Accordingly, the largest connected component of theimage may be determined to be the lip contours. This is possible, inpart because, at the onset of the process, the 2D facial image includedthe mouth region and excluded or cropped out many, most, or all of theother facial features, such as one or more of the chin, jaw, eyes, ears,and most, or all of the nose.

Panel 4750 shows the extracted single pixel width lip contour 7452 alongwith four basic points of the lip contours. The four basic points maycorrespond to the left-most point 4757 of the lip contour 4752, themiddle point of upper lip contour 4755 of the lip contour 4752, theright-most point 4754 of the lip contour 4752, and the middle point oflower lip contour 4756 of the lip contour 4752.

In some embodiments, shortest paths between the left-most point 4754 andthe middle points 4755, 4756 and shortest paths between the right-mostpoint 4754 and the middle points 4755, 4756 are determined, such paths4762 are depicted in panel 4760.

In some embodiments, the shortest paths are refined based on the lipcontours. For example, an iteratively built spline may be formed basedon one or more of the shortest paths and the lip contour 4752. Forexample, an initial point of the spline may be located on the leftcorner of the mouth contour, which may correspond to the left-most point4757. From this initial point a spline is built rightward along thelower lip towards the middle point of the lower lip contour 4756. When,advancement of the spline diverges from lower lip, as determined forexample by maximizing R-square or another method, advancement stops anda new point is placed. The divergence may also be determined based on asum of the square of the distance between the spline and the lipcontours at points along the spline, or another method. The process thenrepeats until one or more splines are built between each of the fourbasic points. In some embodiments, the splines form a closed perimeterthat corresponds to the mouth opening of the facial image.

After the spline or splines 4772 are formed, the splines 4772 may becombined with the facial image, as shown, for example, in panel 4770.

FIG. 13E shows the facial image with the contours 4782 of the 3D toothmodel and lip contours 4784 overlaid thereon. To match the 3D toothmodel tooth contours to the facial image, an expectation-maximization(EM) algorithm is used. First, the 3D tooth model contours are projectedin an image plane. In some embodiments, the image plane is determinedbased on the distance between the imager or camera used to take thefacial image and the focal length of the lens used to take the facialimage. Then for the expectation step of the EM algorithm, for each pixelon the tooth contours from the facial image is matched to a similarpixel found on the 3D tooth model. On the maximization step of the EMalgorithm the jaws or tooth arches of the 3D model are adjusted in oneor more of translation and rotation in one or more of three orthogonaldirections to minimize the total discrepancies between the pixels fromthe facial contours with pixels from the 3D model contours. In someembodiments, camera parameters may also be adjusted minimize totaldiscrepancy. The total discrepancy may be determined based on a sum ofleast squares method or another method. In some embodiments, a pin-holecamera projection may provide an accurate projection of the 3D toothmodel contours.

FIG. 13E shows the aligned lip contours 4784 and 3D model contours 4782within the facial image 4780. The alignment accounts for the portion ofteeth contours 4782 a that are visible though the mouth opening definedby the lip contours 4784 and the portions of teeth contours 4782 b thatare not visible though the mouth opening. When rendering a composite 2Dand 3D image (for example, as shown in FIGS. 12 and 13 ), the visibleportions of the 3D tooth model may be rendered while the nonvisibleportions of the 3D tooth model may not be rendered. For example, theteeth or other portions of the 3D tooth model, also referred to as abite model, obscured by the patient's lips, may not be rendered. Theprocess of forming the composite 2D and 3D image may be performed at oneor more of blocks 540 through 560 of FIGS. 5A-C.

FIGS. 23A-23D show a method 5000 of determining facial landmarks to aidin integrating a 3D tooth model into a facial image of a patient. Atblock 5010 facial landmarks from a facial image are determined. In someembodiments, the facial landmarks are determined based on a neuralnetwork that may have been trained on human faces. FIG. 23B shows afacial image 5060 with identified landmarks. The landmarks includeeyebrow landmarks 5062 that identify the upper edge of a person'seyebrow, eye landmarks 5066 that identify the corners and lids of aperson's eye, nasal landmarks 5068 that identify features of a person'snose, mouth landmarks 5061 that identify features of a person's lips andmouth, and facial outline landmarks 5064 that identity the outline of aperson's face, such as the chin and jaw lines.

At block 5020 the facial midline of the facial image 5060, 5070 isdetermined based on the locations of the facial landmarks. In someembodiments, a subset of facial landmarks are used to determine thefacial midline 5080 of the face in the facial image 5060, 5070. Forexample, in some embodiments, four central landmarks 5072 are used todetermine the facial midline. These central landmarks 5072 are typicallyaligned along the facial midline and include a landmark 5072 a along thenasal ridge between the eyes, a landmark 5072 b at the tip of the nose,landmark 5072 c and the center of the chin, and landmarks 5072 d, 5072 eat the center of the outer lower and upper lips, respectively.

In some embodiments, the average of symmetrical facial landmarks may beused to determine the facial midline. Symmetrical facial landmarks arepairs of landmarks that identify the same feature on each side of theface. For example, the pair of central eyebrow landmarks 5074 include aleft central eyebrow landmark and a right central eyebrow landmark.Similarly, the other eyebrow landmarks may be used, such as the inner5075 and outer 5076 canthus, the ala 5073.

These landmarks are generally equidistant from the facial midline.Accordingly, an average of the location of a pair of symmetrical faciallandmarks is determined based on the location of each symmetricallandmark of the pair. This average location may be used to determine amidline point. The midline points from each pair of symmetrical pointsis then used to determine the facial midline 5080.

In some embodiments, both symmetrical facial landmarks and faciallandmarks 5072 that are generally along the facial midline may be usedto determine the facial midline 5080.

When determining the facial midline using either the facial landmarks5072, the symmetrical landmarks, or a combination of both, the facialmidline may be determined at a straight line that maximizes theR-squared of the distance between the facial midline 5080 and therespective landmarks, such as the landmarks 5072 and the averagelocation of each pair of symmetrical landmarks. In some embodiments, theone or more landmarks may be weighted, such that their location has agreater or lesser effect on the position of the midline. For example,facial landmarks 5072, which may be referred to as midline landmarks maybe weighted greater than symmetrical landmarks.

At block 5030 (see FIG. 23A) a facial plane is determined. A facialplane may be formed based on the facial midline 5080 and the positionand focal length of the imager used to capture the facial image. Thefacial plane may be a 2D plane extending though the facial midline 5080and may be parallel to the sagittal plane of the patient. A facial planemay be used to align a rendering of a 3D model of the patient's teeth,for example in initial, final, or intermediate positions, with thefacial image of the patient.

FIG. 23D shows a 3D model of a patient's teeth 5090 and a facial plane5092. The 3D model includes a dental midline 5094. The dental midlinemay be a line though the midpoint of the patient's arch, for example,the dental midline 5094 may extended between the upper central incisors.In some embodiments, the dental midline 5094 extends in a directionperpendicular to the occlusal plane of the patient. The dental midline5094 may also be determined as described herein with reference to FIG.18 . To align the 3D model 5090 with a facial image of a patient, thefacial plane 5092 and the dental midline 5094 are aligned by rotatingand translating one or more of the 3D model 5090 and the facial plane5092 until the dental midline 5094 is in the facial plane 5092. Therotations and/or translations of the facial plane 5092 are also carriedout on the facial image. At block 5040 (see FIG. 23A), rotations of thefacial image or photo are carried out. In some embodiments, the 3D modelor the facial image may be rotated and translated.

In some embodiments, no rotations or translations of the facial imageare carried out. Instead, the 3D model is rotated or translated untilthe dental midline 5094 is in the facial plane 5092.

At block 5050, after alignment of the facial image and the 3D model,such as described above by aligning the dental midline with the facialplane, the 3D model is inserted into the facial image or image of thepatient.

FIG. 13F shows a process of morphing a rendering from a pre-treatmenttooth arrangement shown in panel 4791 to a post-treatment or plannedfinal tooth arrangement shown in panel 4793. The morphing algorithm mayuse a facial image 4790, the lip contours 4792, and masks of the teethrendered in initial tooth positions 4794 and final tooth positions 4796as inputs. The initial 2D and 3D composite image shown in panel 4791 maybe rendered using the initial 2D facial image and the 3D model of thepatient's teeth rendered in the initial position and the final compositeimage shown in panel 4793 may be rendered using the initial 2D facialimage and the 3D model of the patient's teeth rendered in the finalposition.

To morph between the initial and final tooth positions, a segmentationmask may be built for both the rendering of the teeth in the initialposition and the rendering of the teeth in the final positions. Suchsegmentation masks may include corresponding triangulation such thateach triangle in the rendering of the first position corresponds to atriangle in the final position. The corresponding triangles may, andlikely do, have different shapes in the initial and final renderings.The resulting morph between the teeth in the initial position and theteeth in the final position is a frame-by-frame translation of thetriangles shaped according to the initial position to the trianglesshaped according to the final position with corresponding color andshading translations.

In some embodiments, the incisal edges of teeth may be used to align a3D model of a patient's teeth with a 2D facial image of a patient. Forexample, FIG. 14 depicts an embodiment of the selection or determinationof reference points along the incisal edge or cusps of the teeth on the2D image of a patent. The selection of the reference points 1410, 1420,1430, 1440 may be made according to the process described above withrespect to FIG. 5 -C and block 530. In some embodiments, the referencepoints 1410, 1420 may be the midpoint of the incisal edge of the teeth.In some embodiments, the reference points 1430, 1440 may be a cusp tipof one or more teeth, such as the cusp tips of the canine teeth.

The references points of the 2D image may correspond to similarreference points at the same location on the 3D bite model. FIG. 15depicts the selection of reference points 1510, 1520, 1530, 1540 on the3D bite model of the patient's teeth. The selection of the referencepoints 1510, 1520, 1530, 1540 may be selected according to the processdescribed above with respect to FIGS. 5A-C and block 530. In someembodiments, the reference points 1510, 1520 may be the midpoint of theincisal edge of the teeth. In some embodiments, the reference point1530, 1540 may be a cusp tip of one or more teeth, such as the cusp tipsof the canine teeth.

The integration of the 3D bite model 1040 into the 2D image may proceedas described above with reference to FIGS. 5A-C and block 540. In someembodiments, combining the 3D bite model 1040 with the 2D image 600 ofthe patient includes determining a mouth opening 1010 in the 2D image.In some embodiments, the mouth opening 1010 is the shape of the insideedge of the patient's lips 1020, 1030 in the 2D image. In someembodiments, the portion of the 2D image within the mouth opening 1010is deleted or otherwise removed from the 2D image 600 of the patient. Insome embodiments, the 3D bite model 1040, or a 2D projection of the 3Dbite model 1040 is placed or rendered behind the 2D image 600 such thatthe 3D bite model 1040, or at least a portion of the 3D bite model 1040,is visible through the mouth opening of the 2D image of the patient. The3D bite model 1040 is manipulated and positioned such that the teeth ofthe 3D bite model 1040 are in the same position and the same size as theteeth of the patient in the 2D image 600.

Two corresponding reference points may be considered a pair ofcorresponding reference points. For example, a reference point 1510 atthe left incisal edge of the 3D bite model can correspond with thereference points 1410 at the left incisal edge of the 2D image of thepatient. As additional examples, reference points 1430, 1530 are acorresponding pair of reference points at respective canine cusp tips.

The alignment of the 3D bite model with the 2D image may be performed asdescribed above with reference to FIGS. 5A-C and block 540. The 3D bitemodel 1040 is manipulated and positioned such that the teeth of the 3Dbite model 1040 are in the same position and the same size as the teethof the patient in the 2D image 600. In some embodiments, the 3D bitemodel 1040 is positioned in the 2D image 600 by aligning each of thereference points on the 3D bite model 1040 to each of the correspondingreference points on the 2D image 600 of the patient. In someembodiments, the reference points of the 3D bite model 1040 are alignedwith the corresponding reference points on the 2D image 600 of thepatient by minimizing the sum of the in distance between eachcorresponding pair of reference points. In some embodiments, thereference points of the 3D bite model 1040 are aligned with thecorresponding reference points on the 2D image 600 of the patient byminimizing the sum of the squares of the distance between eachcorresponding pair of reference points. In some embodiments, the 3D bitemodel may be aligned when the distance between all the reference pointsis less than a threshold, such as less than 0.05 mm, 0.1 mm, 0.2 mm, or0.5 mm or wherein the sum of the squares of the distances between thereference points is less than a threshold or minimized.

Virtually Representing a Treatment Outcome Using Automated Detection ofFacial and Dental Reference Objects

In some implementations, one or more modules of the data processingsystem 400 may be configured to virtually represent an orthodontictreatment outcome by detecting and comparing facial and dental referenceobjects against one another. A “reference object,” as used in thiscontext, may refer to one or more physical and/or anatomical features ofa patient that can be used to infer the placement of portions of digital3D models within a 2D image. A “reference object” may be compriseelements of a patient's face or a patient's teeth/dentition. Examples ofa facial reference object include: a facial midline along the sagittalplane of a patient's face; a location of the inferior boarder of theupper lip at a patient's facial midline; a location of the superiorboarder of the lower lip at the facial midline; etc. Examples of adental reference object include: a dental midline; an incisal edgeposition of one or more teeth; a gingival zenith of a central incisor;etc.

In various implementations, the processor(s) 402 gather from the storagesubsystem(s) 406 a 3D model that models an initial position of apatient's teeth. The 3D model may include virtual representations ofeach of the patient's teeth and may depict translations and/or rotationsof the patient's teeth along six degrees of freedom. The 3D model mayinclude virtual representations of the patient's teeth at an initialpre-orthodontic treatment stage as well as along intermediate and/orfinal orthodontic treatment stages.

The camera 425 may be configured to capture an image of the patient'sface. In some implementations, the camera 425 is located remotely to theother modules of the data processing system 400. As an example, thecamera 425 may comprise a dedicated camera incorporated into anintraoral scanner, a cellphone camera, a network-coupled camera, aradiograph, scans related to PVS impressions, etc.

In some implementations, the processor(s) 402 may executecomputer-implemented instructions stored on the storage subsystem 406 toselect facial reference objects on images of the patient's face. Asnoted herein, the facial reference object may include one or more of: afacial midline along the sagittal plane of a patient's face; a locationof the inferior boarder of the upper lip at a patient's facial midline;a location of the superior boarder of the lower lip at the facialmidline; etc. Further, the processor(s) 402 may executecomputer-implemented instructions stored on the storage subsystem 406 toselect dental reference objects on the patient's teeth. The dentalreference objects may be gathered from the 3D model of the patient'sdentition. The processor(s) 402 may further execute computer-implementedinstructions stored on the storage subsystem 406 to assign locations tothe facial reference object(s) and/or the dental reference object(s). Invarious implementations, the processor(s) 402 may executecomputer-implemented instructions stored on the storage subsystem 406 tocompare locations of facial reference objects with locations of dentalreference objects.

Advantageously, the processor(s) 402 may execute computer-implementedinstructions stored on the storage subsystem 406 to modify interimpositions of teeth on the 3D model using comparisons between locationsof facial reference objects and locations of dental reference objects.In some implementations, modifications may be recommended and/orimplemented. Depending on the implementation, the processor(s) 402 mayonly make modifications if a specified threshold value is reached,exceeded, etc.

As an example, the processor(s) 402 may execute computer-implementedinstructions stored on the storage subsystem 406 to modify interimpositions of teeth on the 3D model using comparisons between a facialmidline and a dental midline; to the extent the dental midline does notmatch up with the facial midline, the dental midline of the 3D model maybe modified. As another example, the processor(s) 402 may executecomputer-implemented instructions stored on the storage subsystem 406 tomodify interim positions of teeth on the 3D model using comparisonsbetween an inferior boarder of the upper lip at the facial midline andincisal edge position; to the extent the incisal edge position does notmatch up with the inferior boarder of the upper lip at the facialmidline, the incisal edge position of the 3D model may be modified.

As yet another example, the processor(s) 402 may executecomputer-implemented instructions stored on the storage subsystem 406 tomodify interim positions of teeth on the 3D model using comparisonsbetween an superior boarder of the lower lip at the facial midline andincisal edge position; to the extent the incisal edge position does notmatch up with the superior boarder of the upper lip at the facialmidline, the incisal edge position of the 3D model may be modified. Asyet another example, the processor(s) 402 may executecomputer-implemented instructions stored on the storage subsystem 406 tomodify interim positions of teeth on the 3D model using comparisonsbetween an inferior boarder of the lower lip at the facial midline andgingival zenith of a central incisor; to the extent the gingival zenithof a central incisor does not match up with the inferior boarder of theupper lip at the facial midline, the gingival zenith of a centralincisor of the 3D model may be modified.

In some implementations, the processor(s) 402 may executecomputer-implemented instructions stored on the storage subsystem 406 toidentify a facial type of a patient. A “facial type,” as used herein,may include a class of face shapes and/or characteristics from which toinfer location of dentition after application of an orthodontictreatment plan. The facial type is determined based on a distancebetween the patient's glabella and chin and the distance between thepatient's right and left cheekbone prominences. The processor(s) 402 mayexecute computer-implemented instructions stored on the storagesubsystem 406 to modify widths of specific teeth, such as centralincisors, lateral incisors, and canines in an interim final positionrepresented on the 3D model. In various implementations, theprocessor(s) may execute computer-implemented instructions stored on thestorage subsystem 406 to accommodate the size(s), shape(s), location(s),and relationships of restorative objects, such as crowns, veneers, etc.

FIG. 16A depicts an embodiment of a method 1600 for determining a finalposition of a patient's teeth based, in part, on facial features, toothproportions and positions, and gum positions. The method 1600 may bepart of the process of developing a treatment plan and set of alignersfor a patient as shown and described above, in particular with referenceto FIGS. 1-4 . In some embodiments, method 1600 takes place after theafter an initial treatment planning process as described with referenceto FIGS. 1-4 .

At block 1602 the process of determining the facial midline andinterpupillary line is conducted. FIG. 17 illustrates one embodiment ofselecting, identifying or otherwise determining the facial midline andinterpupillary line of a patient from the 2D image of the patient.Another method is shown and described with respect to FIGS. 23A-D. Insome embodiments, the patient is imaged with a social smile facialexpression. In some embodiments, a point of the subnasion 1730 and theglabella 1710 are selected or otherwise determined. A facial midline1740 is drawn from or through the point of the glabella 1710 to orthrough the point of the subnasion 1730. In some embodiments, the pointof subnasion 1730 is defined, and it remains static. In someembodiments, the image of the patient or the face of the patient in theimage may be in a nonvertical orientation. In such embodiments, theimage of the patient may be rotated until the facial midline 1740 is ina vertical orientation. In some embodiments, the image or face of thepatient in the image may be rotated about either the point of thesubnasion 1730 or the glabella 1710. In some embodiments aninterpupillary line 1760 may be drawn through or between the points ofthe eye pupils 1720 a and 1720 b.

At block 1604 of FIG. 16A, the dental midline from a 2D image of apatient with a social smile is selected or otherwise determined. Asshown in FIG. 18 , the dental midline 1810 may be selected or otherwisedetermined based on the location of the papilla 1820, the incisalembrasure 1830 of the anterior central incisors 1840, or the incisalembrasure between the lower incisors. The papilla 1820 is a smallsounded protuberance on the gingiva and the incisal embrasures 1830 arethe v-shaped valleys between adjacent teeth, such as the two anteriorcentral incisors 1840. The dental midline 1810 may be defined as a linethrough the upper incisal embrasure 1830 of the anterior centralincisors 1840, or the lower incisal embrasure between the lower incisorsand parallel to the facial midline 1740. In some embodiments, a distancebetween the facial midline and the dental midline is determined. In someembodiments, if the distance between the facial midline and the dentalmidline is greater than a threshold value, such as 0.05 mm, 1 mm, 2 mm,3 mm, 4 mm, or 5 mm, the final position of the teeth in the treatmentplan may be determined such that that the dental midline is moved suchthat it is less than the threshold. In some embodiments, the thresholdfor movement is different than the threshold for the final position. Forexample, in some embodiments, the threshold distance below which thetreatment planning does not adjust the dental midline is 2 mm, but oncethe treatment planning determines that the dental midline is moved, thenthe dental midline is shifted to below a second threshold value, such as1 mm or 0.5 mm or until the facial midline and the dental midline arecoincident.

At block 1606 the position of the incisal edge of the upper centralincisors 1960 from a 2D image of a patient with a repose facialexpression is selected or otherwise determined. As shown in FIG. 19 , ahorizontal line perpendicular to the facial midline 1740 at thesubnasion 1743 may be generated or otherwise formed. A lineperpendicular to the facial midline 1740 at the inferior border of theupper lip 1940 (the lower edge of the upper lip) may be generated orotherwise formed. A line perpendicular to the facial midline 1740 at themost inferior incisal edge point of the upper central incisors 1950 (thelowest edge point of the incisors in the upper dental arch) may begenerated or otherwise formed. A line can be drawn perpendicular to thefacial midline 1740 at a target position of the inferior incisal edgepoint of the upper central incisors 1960. The target position may beselected or otherwise determined based on facial aesthetics. An inferiorposition is the direction towards the feet, while a superior position isthe direction towards the head.

Once these lines are generated, the distances between the lines may bemeasured to determine the spatial difference between the existing andthe target positions. In some embodiments, the distance may be measuredrelative to the subnasion 1730 and the inferior incisal edge point ofthe central incisors 1950 to determine the initial position of theincisal edge of the teeth. In some embodiments, the distance may bemeasured between the subnasion 1730 and the target position of theinferior incisal edge point of the upper central incisors 1960, and thisdistance may be used for determining the target position of the incisaledge of the teeth. In some cases, the distances may be measured withreference to the inferior border of the upper lip 1940, rather than thesubnasion 1730. In some embodiments, if the initial position of theincisal edge is outside of a target range, then the incisors may bemoved as part of the treatment planning process to the final targetposition such that the incisal edge is within a target distance range.In some embodiments, the target distance range between the inferiorborder of the upper lip 1940 and the target position 1960 of the incisaledges 1950 may be between 3 and 4 millimeters.

In some embodiments, at block 1606 the position of the incisal edge ofthe upper central incisors 1960 from a 2D image of a patient with asocial smile is determined. As shown in FIG. 20 , a horizontal line maybe generated or otherwise formed perpendicular to the facial midline1740 through the subnasion 1730. A line may be drawn perpendicular tothe facial midline 1740 at the most inferior incisal edge point of theupper central incisors 1950. A line may be drawn perpendicular to thefacial midline 1740 at a target position 1960 of the inferior incisaledge point of the upper central incisors. The point of the superiorborder of the lower lip that intersects the facial midline 2010 may alsobe identified.

The distances between the lines may be measured. In some embodiments,the distances are measured to quantify the spatial difference betweenthe existing and the target positions. In some embodiments, thedistances may be measured between the subnasion 1730 and the inferiorincisal edge point of the central incisors 1950, to determine an initialdistance. In some embodiments, the distance may be measured between thesubnasion 1730 and the target position 1960 of the inferior incisal edgepoint of the upper central incisors, and this distance may be used fordetermining the target position of the teeth as part of the treatmentplanning process. In order to design an aesthetically pleasing smile,the distance between the target position of the inferior incisal edgepoint of the upper central incisors 1960 and the superior border of thelower lip intersecting the facial midline 2010 may be less than or equalto 1 millimeter. In some embodiments, this distance may be less than orequal to 2 mm, for example, when the patient's lower lip is a dynamic orV-shaped lip in a social smile expression. In the treatment planningprocess, the teeth may be moved such that in their final positions, theyare these threshold distances.

At block 1608 the gingival line reference from a 2D image of a patientwith a social smile 2110, 2120 is determined, for example, as shown inFIG. 21 . The location of the gingival zeniths 2140 may be selected orotherwise determined. The inferior border of the upper 1940 can also bemeasured. The distance between the gingival zenith 2140 and the inferiorborder of the upper lip 1940 may be determined based on a compositeimage of the 3D bite model and the 2D image of the patient or from the2D image of the patient without the 3D bite model. In some embodiments,the threshold distance between the gingival zenith and the inferiorborder of the upper lip, below which the gingival zenith may be alteredduring treatment planning may be 3 millimeters. In some embodiments, thetarget range for the final distance after treatment is between −1millimeters (the gingival zenith is above the inferior border of theupper lip and hidden) to 2 millimeters. The central incisor height (CIH)may be measured from the gingival zenith to the edge of the tooth. Anaesthetic CIH can be less than or equal to 12 millimeters. In someembodiments, the treatment plan moves the teeth such that the CIH isless than 12 millimeters.

As described herein, an initial target final orthodontic position of thepatient's teeth may be determined at block 1610 of FIG. 16A. In someembodiments, block 1610 may also include the generation of a treatmentplan based on the initial orthodontic target position.

At block 1612 of FIG. 16A a face type of the patient may be determinedas shown in FIGS. 22 and 23 . As shown in FIG. 22 , reference points andthe distances between pairs of reference points may be selected,measured, or otherwise determined. The location of the glabella 1710 maybe determined at the intersection of the facial midline 1740. A point onthe chin 2240 at the intersection of the facial midline 1740 may also bedetermined. The distance between point on the chin 2240 and the point onthe glabella 1710 are used in conjunction with the distances betweenrespective pairs of the other points to determine the shape of thepatient's face.

A respective pair of points, one on each of the cheekbones 2210 a, 2210b and the distance between them can be determined. A respective pair ofpoints, one on each of the mandibular angles 2230 a, 2230 b and thedistance between them can be determined. A respective pair of points,one on each of the temples 2220 a, 2230 b and the distance between themcan be determined. Once these respective pairs of points and thedistances between them are measured or otherwise determined, thedistances between the points may aid in determining the spatial facialprofile and to categorize the facial type as shown in FIG. 23 .

FIG. 23 shows three face types, a short face type 2310, and average facetype 2320, and a tall face type 2330. In a short-type face 2310, thedistance between glabella 1710 and the chin 2240 is similar to or equalto the distance between the cheekbones 2210 a and 2210 b and similar orequal to the distance between the mandibular angles 2230 a and 2230 b,for example, the distances are within 10% of each other.

In an average-type face 2320, the distance between the glabella 1710 andthe chin 2240 is much greater than the distance between the cheekbones2210 a and 2210 b, for example between 15% and 20%; the distance betweenthe cheekbones 2210 a and 2210 b is similar to or equal to the distancebetween the mandibular angles 2230 a and 2230 b; and the distancebetween the temples superior to the ears 2220 a and 2220 b is greaterthan the distance between the mandibular angles 2230 a and 2230 b, forexample, between 10% and 15%.

In a tall-type face 2330, the distance between the glabella 1710 and thechin 2240 is much greater than the distance between the cheekbones 2210a and 2210 b, for example greater than 20%; the distance between theglabella 1710 and the chin 2240 is much greater than the distancebetween the mandibular angles 2230 a and 2230 b, for example greaterthan 20%; and the distance between the glabella 1710 and the chin 2240is much greater than the distance between the temples superior to theears 2220 a and 2220 b, for example greater than 20%, while thedistances between the temples superior to the ears 2220 a, 2220 b, thedistance between the mandibular angle 2230 a, 2230 b, and the distancebetween the cheekbone prominence 2210 a, 2210 b, are all similar, forexample, within 10%.

At block 1614 the inter-canine width (ICW) is determined, for example,as shown in FIG. 24 . The line that demarcates the inter-canine width2420 a and 2420 b may be selected or otherwise determined by verticallines through the most distal points on the buccal surfaces of the uppercanines 2410 a and 2410 b. These ICW lines 2420 a and 2420 b aredetermined based on the teeth 2430 being in the orthodontic finalposition of the 3D bite model, as shown in FIG. 24 . These ICW lines2420 a and 2420 b are parallel to the dental midline 1810 and intersectthe line drawn between the upper canines 2410 a and 2410 b.

At block 1616 the proportions of the upper anterior tooth widths may bedetermined. In some embodiments, the proportions of the upper anteriortooth widths may be determined based on the Recurring Esthetic Dental(RED) proportion or the upper anterior tooth width. Under REDproportions, the successive width proportion when viewed from the facialaspect should remain constant from the midline toward the posterior forthe six anterior teeth between and including the canines. This propertyoffers great flexibility to match tooth properties with facialproportions. Table 1 shows the RED proportions and anterior total widthsbased on the inter-canine-width (ICW) of the patient and the patient'sface type. During the treatment planning process the teeth may be movedand/or restorative objections may be used such that at the end oftreatment, the patient's teeth are within the RED proportions.

TABLE 1 Calculating RED Proportion & Anterior Total Widths fromInter-Canine Width (ICW) and Face Type Desired RED Proportion AnteriorTooth Widths Face Central Incisor Lateral Incisor Canine Type RED %Width (CIW) Width (LIW) Width (CW) Tall 66% RED ICW/4.2 CIW/0.66LIW/0.66 Average 70% RED ICW/4.4 CIW/0.7 LIW/0.7 Short 75% RED ICW/4.6LIW/0.75 LIW/0.75

At block 1618 the central incisor proportion (CIP), which can bedetermined from the ratio of the central incisor width (CIW) or thewidth 2520 of the central incisor 2500 and the central incisor height(CIH) 2510 or the height of the central incisor 2500, is determined. Ina short-type face 2530, the CIP is less than or equal to 65%. In anaverage-type face 2540, the CIP is 78%. In a tall-type face 2550, theCIP is greater than or equal to 85%.

At block 1619 the real gingival line is determined, as shown in FIG. 26. The location of the final position of gingival zeniths 2140 can bedetermined. The inferior border of the upper 1940 can also bedetermined. The distance between the gingival zenith 2140 and theinferior border of the upper lip 1940 may also be determined. Thecalculated distances can be compared to acceptable gingival distances,as discussed above, and the final orthodontic position of the patient'steeth may be adjusted such that the distance is within a target range orbelow a target value.

At block 1620 restorations from a tooth library are placed on the 3Dbite model. FIG. 27 shows restorations 2730 from the tooth library 2710that can be placed on the 3D bite model for creating the final 3D bitemodel. The initial reference points for the placement process caninclude, but are not limited to, the incisal edge 2720, the gingivalline 2710, the line that demarcates the inter-canine width 2420 a and2420 b at the most distal points on the buccal surfaces of the uppercanines, RED percentages, and central incisor proportions. In someembodiments, the 3D tooth models from the tooth library are idealized orgeneric shapes of teeth. The shapes of these tooth models may be appliedto the 3D bite model of the patient to aid in determining the shape of arestorative object to be used on the patient's teeth.

At block 1622 the smile arc of the patient's teeth in the finalorthodontic position with the restorative shapes is determined. As shownin FIG. 28 , the following reference points may be determined: thecentral incisor edge 2810 a and 2810 b, the canine cusp 2820 a and 2820b, and the lip commissures 2830 a, 2830 b. The teeth from the toothlibrary may be modified such that the tooth surface shapes of the 3Dbite model 2840 match or more closely match the target reference pointsfor the central incisor edge 2810 a and 2810 b, the canine cusp 2820 aand 2820 b, and the lip commissures 2830 a, 2830 b. Then adjustments aremade on the canine tooth such that the length of the central incisor issimilar to or equal to the length of the canine. Reference points on thecentral incisor edge 2810 a and 2810 b and the canine cusp 2820 a and2820 b can be adjusted via the canine. The arc of the smile can becreated with a line intersecting the lip commissure 2830 a, the caninecusp 2820 a, the central incisor 2810 a, the central incisor 2810 b, thecanine cusp 2820 b, and the lip commissure 2830 b.

At block 1624 tooth inclination may be determined and modified. As shownin FIG. 29 , reference lines for tooth inclination may be formed aslines extending between or though the gingival zenith 2910 and a centerpoint incisal edge points 2920 or tooth cusps. These reference lines maybe used to determine the angle of the tooth to with respect to thedental midline. In some embodiments, the shape or position of the finaltarget positions of the teeth may be adjusted such that the angles ofeach reference line with the dental midline increases for each toothaway from the dental midline. For example, the angle of the centralincisor line is less than the angle of the reference line for thelateral incisor, which is less than the angle of the canine.

At block 1626 a final or revised final orthodontic position of the teethis determined and a treatment plan may be regenerated. In someembodiments, the positions of the teeth may be generated and displayedas shown in FIG. 30 , which illustrates the 3D bite model 3020integrated with a facial image of the patient 3010. At block 1628, thefinal orthodontic and restorative positions of the teeth are evaluatedand feedback may be provided for further revisions to the final positionand the treatment plan. FIG. 31 illustrates the 3D bite model 3120 afterimplementation of the treatment plan. The treatment plan can include anevaluation of the restorative overlay, a tooth mass reductionassessment, and an assessment of restorative option versus orthodonticsolutions. The final composite image 3110 can include the integratedview of the 2D image of the patient and the 3D bite model. The finalcomposite image 3110 and the final 3D bite model 3120 can be showcasedto the patient.

FIG. 16B depicts a method 1600B for virtually representing a treatmentoutcome using automated detection of facial and dental referenceobjects, according to some embodiments. Some or all of the operations inthe method 1600B may overlap with method operations discussed in thecontext of FIG. 16A. At an operation 1640, an initial position of thepatient teeth may be identified using a 3D representation of thepatient's teeth. The initial position may include the patient's teeth atan initial or at an intermediate stage of an orthodontic treatment plan.

At an operation 1642, an estimated interim final orthodontic position ofthe patient's teeth may be determined using a virtual representation ofone or more force systems applied to the patient's teeth. In variousimplementations, application of virtual representations of forces and/ortorques used as part of the orthodontic treatment plan may be modeledagainst the 3D representation of the patient's teeth.

At an operation 1644, one or more images of the patient's face may begathered. These images may be gathered from a camera or other imagecapture device as discussed further herein.

At an operation 1646, one or more facial reference objects on the imageof the patient's face may be identified. The one or more facialreference objects may correspond to a physical or anatomical featureproviding a first reference position to the patient's face. In someimplementations, a geometrical analysis of the image of the patient'sface may be performed, where desired RED proportions and/or anteriortooth widths are determined. The patient's “face type” may bedetermined.

At an operation 1648, one or more dental reference objects on the first3D representation of the patient's teeth may be identified. The dentalreference objects may correspond to physical or anatomical featureproviding a second reference position to the patient's dentition.

At an operation 1650, a relationship between the one or more facialreference objects and the one or more dental reference objects may beidentified. The relationship may involve measurement of distancesbetween the facial reference object(s) and the dental referenceobject(s). In some implementations, the relationship may involvealignment of the facial reference object(s) and the dental referenceobject(s).

At an operation 1652, the estimated interim final orthodontic positionof the patient's teeth in the first 3D representation may be modifiedbased on the relationship between the one or more facial referenceobjects and the one or more dental reference objects. In someimplementations, the modification may only occur if the relationshipmeets or is not greater than a specified threshold. As an example, themodification may only occur if a distance between the facial referenceobject(s) and the dental reference object(s) meets or does not exceed aspecified distance threshold.

At an operation 1654, instructions to integrate a modified estimatedinterim final orthodontic position based on the relationship between theone or more facial reference objects and the one or more dentalreference objects may be provided. In some implementations, the facialreference objects may be used as the basis of a target for the dentalreference objects. Examples include setting a dental midline thatmatches a facial midline; setting an incisal edge position that matchesa location of the inferior boarder of the upper lip at the facialmidline; setting an incisal edge position that matches a location of thesuperior boarder of the upper lip at the facial midline; and setting agingival zenith of a central incisor that aligns with the inferiorboarder of the upper lip at the facial midline. At an operation 1656,instructions to design and/or manufacture an orthodontic appliance usingthe modified estimated interim final orthodontic position are provided.

Matching a Bite Model with a 2D Image

FIG. 32 depicts a composite image 3200 of a 2D image of the face of aperson 3210 and a 2D image of a 3D bite model 3220. In the 2D image ofthe patient, the patient can have an open mouth. The opening of themouth 3230 is demarcated by edge of the lips 3214. The teeth 3260 andthe gingiva 3250 of the 3D bite model 3220 can be shown in the openingof the mouth 3230.

The 3D bite model 3220 can be a 3D digital visualization of the teeth,and it can be used to visualize the shape and position of the teeth in acurrent state or in a future state wherein the future state can be theshape and position of the teeth during or after undergoing a treatmentplan to reposition the teeth of the patient. The treatment plan caninclude orthodontic measures for correcting malocclusions and otherorthodontic issues with the patient's teeth, for example correcting anoverbite, an underbite, tooth rotation, or tooth tipping. In someembodiments, the treatment plan can include restorative measuresincluding but not limited to installing a crown, a veneer, or bridge.

The composite image 3200 can be used to provide facial context to thevisualization of the 3D bite model. In some embodiments, the compositeimage 3200 provides facial contextual visualization of the 3D model ofthe shape and position of the patient's natural teeth. In someembodiments, the composite image can provide facial visualization of the3D model of the shape and position of the teeth undergoing the treatmentplan. Such contextual information aids in the evaluation of the dentaltreatment plan and in determining the final position of the patient'steeth, for example when used with the treatment planning processes andsystems described herein. Contextual information also aids in thepatient's understanding of the treatment plan and in making an informedtreatment decision.

A composite image of a 2D image of the face of a person and a 2D imageof a 3D bite model (3D bite model) can create an unnatural look. Thisunnatural look may be described by the uncanny valley effect, which is ahypothesis that describes the adverse human emotional response to humanreplicas. Human replicas that appear nearly, but not quite like humanbeings elicit feelings of eeriness and revulsion. In contrast, humanreplicas that are distinctly different than humans do not elicit thisnegative emotional response. Composite images that include 3D bitemodels can seem almost, but not entirely, human when viewed in thecontext of a natural human face. Therefore, some composite images elicitfeelings of unease in the dental professional or in the patient. In someembodiments, an unnatural look can be caused by aesthetic differencesbetween the natural teeth and the 3D bite model, including but notlimited to, 3D teeth features, blur effect, teeth color, gingiva color,intensity of whiteness, intensity of red color, intensity of greencolor, and intensity of blue color. In order to aid in reducing theunpleasant reactions, the composite can be made to have a more seamlessintegration of the 3D bite model into the 2D image of the patient. Asdescribed below, systems and methods may be implemented to aid inmatching the 3D bite model with the 2D image of a patient. Variousfeatures of the 3D bite model can be controlled to aid this matching.Some embodiments of matching a 3D bite model with a facial image may useneural networks or machine learning algorithms, as shown and describedwith respect to FIGS. 36A and 36B.

FIG. 33 illustrates a controller 3300 that may be a digital userinterface for adjusting features of the 3D bite model 3220. Thecontroller 3300 may be part of the system 400 shown and described abovewith reference to FIG. 4 . The controller 3300 may include toggleswitches 3310, 3311, 3312, 3318 and scaling bars that aid in adjustingfeatures of the 3D bite model 3220. The controller 3300 can be used tocontrol one or more of a plurality of features of the 3D bite modelimage. One of a plurality of controllable features can be to adjust the3D tooth effect, and the 3D tooth effect can be turned on or otherwiseadjusted via, for example, a switch, such as the 3D teeth feature toggleswitch 3310. One of a plurality of controllable features can be tocreate a blur effect, and the blur effect can be turned on or otherwiseadjusted via a switch, such as the blur model toggle switch 3311. One ofa plurality of controllable features can be used to adjust teeth color,and the teeth color can be adjusted by or otherwise adjusted via aswitch, such as the adjust teeth color toggle switch 3312. One of aplurality of controllable features can be used to adjust teeth coloroptions (indicated by the pick color bar 3313) using the press-buttonlocated in bar 3319 a dental professional may select a location on the2D image for use in coloring the 3D bite model. The tool color may alsobe adjusted using preselected color options, which may be standarddental colors. For example, the color options may include an A1 color3319, a B1 color 3320, a C1 color 3321, a D2 color 3322, and anautomatic color 3323 chosen, for example, based on the colors of theteeth of the 2D image of the patient. One of a plurality of controllablefeatures can be to adjust the intensity of whiteness, and the intensityof whiteness can be adjusted or otherwise selected by moving the slider3324 along the scale 3324 of the whiteness adjuster 3314.

One of a plurality of controllable features can also include adjustmentsand selection of the red, green, and blue color balance of the teeth ofthe 3D bite model. The controller 3300 includes a red adjuster 3325, agreen adjuster 3326, and a blue adjuster 3327 in respective colorselectors 3315, 3316, 3317. The adjusters and selectors may change thecolor intensity or balance of one both of the teeth and gingiva. Asshown in FIG. 33 , the color adjusters and selectors adjust the colorbalance of the gingiva.

A quality metric, such as an integration quality metric, mayquantitatively describe the difference or degree of agreement between anintegrated composite image and a 2D image of a patient. In someembodiments, the quality metric is defined by a scale that correspondsto the degree of agreement or difference between the integratedcomposite image or the 3D bite model of the integrated composite imageand the 2D image of the patient or a portion of the 2D image of apatient, such as the image of the teeth and gingiva of the patient. Thescale of the quality metric may be a closed or open ended range. Forexample, in an open ended range the metric may start at 0, indicating aperfect match between the composite image and the image of the patient,and increase without limit based on the degree of agreement, withincreasingly high numbers indicating increasingly poor match oragreement between the images. In a closed ended range 0 may indicate aperfect match while 1, 10, or 100, may indicate a complete mismatchbetween the images or a mismatch above a predetermined threshold betweenthe images.

FIG. 34 illustrates a 3D bite model in the composite view with variousdegrees of match between the respective composite view 3410, 3420, 3430and a reference 2D image of the patient. The matching may be performedwith the controller, for example as described above with reference toFIG. 33 or according to the method 3500 described below with referenceto FIG. 35 , or another method, such as that shown and described withreference to FIGS. 36A and 36B. The composite image 3410 shows anexample of a composite image with a high degree of mismatch and lack ofagreement with a reference image. In such an image, the mouth opening3411, delineated by the inner edges of the lips 3412 bear little to noresemblance with the mouth, teeth, and gingiva of an image of thepatient. The quality metric of such a composite image may be high, suchas greater than a threshold value.

The composite image 3420 shows an example of a composite image with amoderate degree of mismatch and lack of agreement with a referenceimage. In such an image, the mouth opening 3411, delineated by the inneredges of the lips 3412 includes a colored 3D bite model 3425 of thepatient within the mouth opening 3411 with basic color and shading addedsuch that the teeth 3424 and gingiva 3423 are apparent, but significantand noticeable differences may still exist between the 2D image of thepatient. The quality metric of such a composite image may be high, suchas greater than a threshold value, but lower than that of the compositeimage 3410.

The composite image 3430 shows an example of a composite image with alow degree of mismatch and high degree of agreement with a referenceimage. In such an image, the mouth opening 3411, delineated by the inneredges of the lips 3412 includes a colored 3D bite model 3435 of thepatient within the mouth opening 3411 with accurate color and shadingadded such that the teeth 3434 and gingiva 3433 include few, if any,noticeable differences between the 3D bite model 3435 and the image ofthe patient. The quality metric of such a composite image may be low,such as zero, or less than a threshold value of acceptable agreement.

FIGS. 35 and 36 illustrate a method of matching the 3D bite model withthe 2D image of a patient. The process for obtaining the 3D bite modelin integration view 3500 can be based on a comparative analysis wherebythere is a minimization of the differences between a first image and asecond image. The minimization process may reduce the quality metric toat least a threshold value of acceptance. FIG. 36 shows a reference 2Dimage 3641 of the patient, including their natural teeth 3642 andnatural gingiva 3643, along with three images of a composite image,including a composite image 3610 at an initial stage, a composite image120 at an intermediate stage of processing, and a composite image 3630at a final stage of processing.

The process for obtaining the 3D bite model 3500 may include multiplesteps and may be iterative. At block 3510 the default settings for therendering parameters are set. Such default parameters may serve as theinitial conditions of the matching process. The rendering parameters canbe derived from the parameters of natural teeth 3642 and natural gums3643 in the 2D image 3641. Subsequently, as depicted in block 3520, thevalue of the integration quality metric can be determined by comparingthe composite image to the 2D image of the patient. The comparisonbetween the composite image 3610 and the 2D image 3641 of the patientmay include a pixel by pixel comparison of the differences between thepixels in the composite image and the corresponding pixels in the samelocation in the 2D image of the patient.

In some embodiments, the quality metric, determined based on thedifferences in the pixels, may be the Mean Absolute Error (MAE) of thepixels compared between the two images. In some embodiments, the qualitymetric, determined based on the differences in the pixels, may be theMean Squared Error (MSE) of the pixels compared between the two images.The difference in the pixels may include the difference between one ormore of the red, green, blue, and luminance values of each pixel. Insome embodiments, the quality metric may be based on a peaksignal-to-noise ratio, such as the PSNR-HVS-M or PSNR-HA methods.

In some embodiments, other methods for calculating the quality metricmay be used. For example, databases of scored images may be used in thequality metric, for example, a metric comparison using the TID2008,TID2013, LIVE, Toyama, IVC, CSIQ, and others may be used. In someembodiments, perceptual visual quality metrics such as FSIMs, SSF,PSNR-HAc, SR-SIM, BMMF, and others may be used. The metric comparisonused in calculating the quality metric may include the use of Spearman'scorrelation, the Kendall correlation, or others.

In block 3530 a step of minimization may be performed where theparameters, such as the rendering parameters, of the 3D bite model areadjusted to aid in reducing the quality metric to below a thresholdvalue. The parameters may include teeth color, gingival color, materialproperties and light properties. Color parameters may include colortemperature, red, green, and blue color intensities, saturation,luminance of the teeth and gingiva. Material parameters may includesurface texture modifications such as surface textures and surfacereflectivity. Light parameters may include shading (such as thevariation of shading on the teeth between the front teeth, which areexposed more directly to light, and the back teeth, which are lessexposed to outside light due to the light blockage by the lips andcheeks), light source location, and type, such as a point source ornon-point source of light. The process alters these parameters in such away as to minimize the quality index. In some embodiments, as part ofthe minimization process, a Monte Carlo method may be implementedwherein various parameters are modified and an estimation of theireffect on the quality index is determined. The parameters of the 3D bitemodel may then be modified based on the results of the Monte Carloanalysis.

At block 3540 the new value of the integration quality metric may becalculated based on the image of the 3D bite model generated using therevised parameters. The interim composite image 3620 shows the interimintegrated 3D bite model 3621 with the interim parameters applied. Atblock 3550 the quality metric is compared to a threshold value. If thequality metric is acceptable based on the threshold value, then theprocess proceeds to block 3560. The quality metric may be acceptable ifthe value of the quality metric is below the threshold value. If thequality metric is not acceptable based on the threshold value, thenanother step of minimization and calculation of the quality metric maybe performed as in blocks 3530 and 3540. As a result of this process,the 3D bite model may more closely match that of the image of thepatient. In some embodiments, the 3D bite model may have parameters thatare similar to or the same as the parameters of the 2D image 3641. Afinal composite image 3630 may be generated based on applying theparameters determined earlier in the process to the 3D bite model 3631such that the 3D bite model 3631 is matched to the colors of the naturalteeth 3642 of the 2D image 3641 of the patient. Such a process mayresult in a seamless and natural integration of the 3D bite model intothe image of the patient.

In some embodiments, neural networks, such as generative adversarialnetworks or conditional generative adversarial networks may be used tointegrate a 3D model of teeth in a final position with a facial image ofa patient and match the colors, tones, shading, and other aspects of the3D model with a facial photo. FIG. 36A depicts an embodiment of a method4800 of integrating a 3D model of a patient's teeth in a clinical finalposition with a facial image of a patient. At block 4802, the neuralnetwork is trained using facial images. In some embodiments, the facialimages may include images of peoples' faces having a social smiles. Insome embodiments, the facial images may include facial images ofpatient's teeth before orthodontic treatment. During training, patient'steeth and their contours may be identified. For example, each tooth maybe identified by type (e.g., upper left central incisor, lower rightcanine). Other aspects and features of the image may also be identifiedduring training, such as the location and color of the gingiva, thecolor of the teeth, the relative brightness of the surfaces within themouth, and others.

Referring to FIGS. 36A and 36B, after training, the neural networkreceives inputs at block 4804 for use in generating a realisticrendering of the patient's teeth in a clinical final position. In someembodiments, the inputs may include one or more of an image of arendering of a 3D model of the patient's teeth in a clinical finalposition or a 3D rendered model of the patients teeth in the clinicalfinal position 4808, the clinical final position determined, forexample, according to an orthodontic treatment plan, a blurred initialimage of the patient's teeth 4810, and a color coded image 4812 of the3D model of the patient's teeth in the clinical final position.

The image of a rendering of a 3D model of the patient's teeth in aclinical final position or the 3D rendered model of the patient's teethin the clinical final position 4808 may be determined based on theclinical orthodontic treatment plan for moving the patient's teeth fromthe initial position towards the final position, as described above. Theimage or rendering 4808 may be generated based on the imagingperspectives determined as described with respect to FIG. 13E. Forexample, one or more of the imaging distance, the focal length of theimaging system, and the size of the patient's teeth in the initialfacial image may be used to generate the image or rendering.

The blurred image of the patient's teeth 4810 may be generated using oneor more blur algorithms, such as a Gaussian blur algorithm. The Gaussianblur preferably has a high radius, for example, a radius of at least 5,10, 20, 40, or 50 pixels. In some embodiments, the blur is sufficientlygreat that the tooth structure is no longer readily apparent to a humanobserver.

The coded model of the patient's teeth 4812 may be a red-green-blue(RGB) color coded image of a model of the patient's teeth, with eachcolor channel corresponding to a different quality or feature of themodel for each tooth 4814 of the teeth 4813 of the patient. For example,the green color channel, which may be an 8-bit color channel indicatesthe brightness of the blurred image 4810 on a scale of 0 to 255 as, forexample, overlaid on the 3D model.

The red color channel may be used to differential each tooth and thegingiva from each other. In such an embodiment, the gingiva may have ared channel value of 1, the left upper central incisor may have a redvalue of 2, the right lower canine may have a red channel of 3, theportions of the model that are not teeth or gingiva might have a redchannel value of 0, and so on, so that the red channel value of eachpixel identifies the dental anatomy associated with the pixel.

The blue color channel may be used to identify the angle of the teethand/or gingiva with respect to the facial plane. For example, at eachpixel location the angle normal of the surface of the dental structure,is determined and a value between 0-255 (for 8-bit color channels) isassigned to the pixel. Such information allows the neural network to,for example, model light reflectivity from the dental surfaces.

At block 4806 the neural network uses the inputs and its training torender a image realistic image of the patient's teeth in a finalposition. This photo realistic image is then integrated into the mouthopening of the facial image and an alpha channel blurring is applied.

The method 4800 provides a significantly more realistic integration ofthe 3D model with the facial image than previous methods. This pushesthe realism beyond the uncanny valley.

Tooth Mass Reduction for Restorative Objects

Patients can display symptoms of malformed teeth or injured teeth.Malformed or injured teeth may be chipped, broken, worn down throughgrinding or other means, or simply malformed.

A restorative object can be used to treat one or more injured ormalformed teeth. A restorative object, such as an artificial tooth orartificial part of the tooth, restores or corrects the shape of aninjured or malformed tooth. In some embodiments, a restorative objectmay be a crown, which is a tooth-shaped cap that is placed over a toothto restore its shape and size and improve its appearance. In someembodiments, the restorative object may be a veneer, which is a thinshell of material that covers the front surface of a tooth. Therestorative object can have a predetermined shape and size. In order toaid the fitting of the restorative object over the tooth, the tooth mayundergo tooth mass reduction where a portion of the tooth is removed toprovide a mounting surface to receive the restorative object.

Different processes for applying restorative objects may use differingamounts of tooth reduction. For example, applying a veneer to a toothmay result in less tooth mass reduction than applying a crown. Differingamounts of tooth mass reduction may also be seen between differentapplications of similar restorative objects. For example, applying aveneer to a tooth in a first position may involve removing more toothmass than applying a veneer to the same tooth, but in a differentposition. Evaluating tooth mass reduction for restorative objects indifferent positions may aid in reducing or minimizing the amount oftooth mass loss due to the application of restorative objects.

FIG. 37 depicts an illustration of teeth with restorative objectsapplied thereon. In some embodiments, such as those shown in FIG. 37 ,the root 3710 and 3720 of the tooth extending to the neck 3711 and 3721may remain intact. As shown in FIG. 37 , a prosthetic crown 3713 may beapplied to a reduced tooth 3712 or a veneer 3714 may be applied to areduced tooth 3722.

FIG. 38 depicts how the amount to tooth mass loss is determined. To aidin fitting the restorative object to a tooth, the tooth may undergotooth mass (or volume) reduction where a portion of the tooth is removedand the remaining tooth is a tooth with a reduced mass or volume. Asshown in FIG. 38 the crown 3832 of the tooth is chipped 3842 orotherwise broken. The broken line represents the reduced portion of thecrown 3840 after tooth mass reduction. In this embodiment, the remainingreduced portion 3840 serves as the mount for the restorative object,such as a prosthetic crown. The mass or volume 3846 removed from thetooth in preparing the tooth for the restorative object is determined,for example, by subtracting the volume of the reduced crown 3840 fromthe volume of the natural crown 3842.

In some embodiments, the removed tooth mass is healthy tooth mass but isremoved in order to create space for the addition of the restorativeobject. It may be beneficial to the patient if the amount of removedtooth can be reduced or minimized. The amount of tooth mass that isremoved can be determined with a 3D bite model during the treatmentplanning process. The position and shape of the restorative object canbe visualized with the integration of the restorative object on the 3Dbite model.

FIG. 39 illustrates a 3D bite model 3900 that depicts teeth, gingiva3940, a restorative object on the right canine 3910, a restorativeobject on the right lateral incisor 3920, and a restorative object onthe right central incisor 3930. The visualization of the restorativeobject in the 3D bite model may be useful in determining the shape orposition of the restorative object on the patient's natural teeth. Theability to determine the shape of the restorative object before dentalwork, such as orthodontic corrections, may allow for tooth massreduction analysis, where the amount of tooth loss is determined forvarious restorative object shapes on teeth in one or more finalorthodontic positions. The restorative object shape that uses the leasttooth mass loss may be beneficial to the patient.

Dental professionals and technicians may modify one or both of therestorative object's placement and the orthodontic final teeth positionsto minimize or reduce tooth mass reduction. The 3D bite model can beused to determine a movement a tooth of a patient from a first positionand orientation to a second position and orientation as describedherein. Computer-based treatment planning can be used to facilitate thedesign of each tooth trajectory in the treatment plan, as describe abovewith reference to FIGS. 1-4 . During the treatment planning process theposition and shape of the restorative object may be modified to matchthe treatment goals. As shown in FIG. 40 , the process 4000 forobtaining the 3D bite model may begin with determining an initial toothposition 4010, such as an initial position for one or more teeth of thepatient. In some embodiments, the initial positions of the teeth aredetermined as describe above with reference to FIGS. 1-4 . At block4020, the interim final orthodontic positions of the teeth aredetermined, for example as described above with reference to FIGS. 1-4 .

At block 4030 the interim restorative object position is determined. Theprocess for determining an interim restorative object position 4030 maybegin by adjusting the orthodontic position to an interim orthodonticposition at block 4031 and then determining at new interim restorativeposition at block 4032. The tooth mass loss of the new interimorthodontic position is compared to the tooth mass loss of one or moreof the previous interim orthodontic and restorative positions at block4033 wherein the new interim restorative position is evaluated. At block4040 the final orthodontic position and the restorative object positionis determined. In some embodiments, the final orthodontic position andthe restorative object position is determined by selecting the finalteeth positions that minimize tooth mass or volume loss among theevaluated positions. In some embodiments, the positions may bedetermined based on minimaxing certain restorative procedures, such asminimizing the number of crowns. In some embodiments, less invasiveprocedures, such as the use of veneers, are given priority over moreinvasive restorative procedures, such as crowns and root canals or wholetooth extraction and prosthetics. This process allows for the evaluationof a new restorative position at each stage of the orthodontic treatmentplan. The various interim restorative positions provide multiple optionsduring the orthodontic treatment plan of when to apply the restorativeobject to the patient. The amount of tooth mass reduction can bedifferent at each interim orthodontic position, thus providing moreoptions.

Restorative Object Position on 3D Models

The restorative object is a part of the final 3D bite model and can bemodified to reach the treatment goals. One treatment goal may be theaesthetic and functional placement of the teeth. For example, in someembodiments, the final or target position of the teeth in a treatmentplan may include positioning and shaping the teeth such that theirshapes and positions conform to Recurring Esthetic Dental (RED)proportion. The dimensions of the boxes of the proportion widget aredetermined by the Recurring Esthetic Dental (RED) proportion. Under REDproportions, the successive width proportion when viewed from the facialaspect should remain constant from the midline toward the posterior forthe six anterior teeth between and including the canines. This propertyoffers great flexibility to match tooth properties with facialproportions. RED proportion may aid in determining the final shape andsize of one or more restorative objects and the final or target positionof the patient's teeth. The 3D bite model can be assessed by determiningthe various dimensions of the patient's teeth, for example the widget4260 illustrates the demarcation of the shape the teeth. FIG. 42illustrates the various dimensions and positions of the teeth 4240 andgingiva 4215 that may be determined. The dimensions of the medialincisor 4211, the lateral incisor 4221, and the canine 4231 may bemeasured. These dimensions may include the width of the medial incisor4210, the width of the lateral incisor 4220, and the width of the canine4230. Also, the intercanine width (ICW) may be determined based on thedistance between the right and the left canine 4231. The height of eachtooth 4250 can also be determined from the 3D bite model. The initialdimensions of the teeth are determined based on the initial 3D bitemodel built from the scan of the patient's teeth and using the midlineplane and occlusal plane.

FIG. 43 illustrates a 3D bite model of the upper jaw from both theanterior position on the left and from the occlusal place, on the right.

FIG. 44 illustrates the modified shape and position of the restorativeobjects on a 3D bite model 4400. The final shape and position of therestorative object for each tooth of a plurality of teeth may be shapedaccording to the RED proportion as shown by the bounding boxes 4460. Thewidth of the central incisor 4410, lateral incisor 4430, and the canine4440 are indicated by the bounding boxes. Similarly, the height of thecentral incisor 4450, lateral incisor 4452, and the canine 4454 areindicated by the bounding boxes.

FIG. 45 illustrates a virtual depiction 4500 of an orthodontic treatmentof a patient. The visual depiction can be generated in accordance withembodiments disclosed herein, such as the methods disclosed above inFIGS. 5A-C. The visual depiction can be a composite image, or modifiedimage, with a first portion of the image 4510 corresponding to an imageof the patient, such as a photograph, and a second portion of the image4520 corresponding to a 3D model of the patient's dentition at a stageof a treatment plan. The two portions are generated by combining the 3Dmodel with the image in accordance with embodiments disclosed herein,thereby generating a composite image showing the patient's face (or atleast a portion thereof) in combination with the patient's dentition (orat least a portion thereof) at a stage of the treatment plan. Aninterface can be provided to allow the displayed composite image to bechanged; for example, to change from a first stage of treatment to asecond stage of treatment. The interface can include a user inputdevice, such as a mouse or touchscreen, for example. The interface caninclude a button, scrollbar, or other method of indicating a desiredchange. For example, FIG. 45 illustrates a scrollbar 4530 operable tochange between stages of a treatment plan, thereby showing changes to apatient's smile as predicted over the course of the plan. Providing anaccurate, photo-realistic view of how a patient's dentition will beorthodontically improved over the course of treatment significantlyimproves patient understanding of the benefits of orthodontic treatment.In some instances, an appointment reminder can be sent to the patient,such as by email, including one or more of the modified images. Themodified image could show patient's face as predicted at the end of thetreatment plan, for example, or the patient's face as predicted afterthe next stage of treatment. The image can also be presented in the formof a video file, illustrating the change of the patient's dentition overthe course of treatment. The video file can be generated and shown tothe patient during an initial exam, for example, and can alternativelyor additionally be sent to the patient in one or more reminder emailsfor orthodontic appointments.

FIG. 46 illustrates a comparative display 4600, showing a virtualdepiction in comparison to an initial image of the patient. A modifiedimage 4610 of the patient can be shown adjacent to an initial image 4620of the patient. The modified and initial images can be shown in adjacentwindows or frames of a computer display, for example. The modified image4610 can be generated in accordance with the embodiments describedherein. For example, the modified image 4610 can correspond to thevisual depiction 4500 illustrated in FIG. 45 . Similar features to thosedisclosed with FIG. 45 can be included in the modified image 4610, suchas a scrollbar or other interface and an optional video illustration ofpredicted progress during a treatment plan. The image 4620 of thepatient can be a photograph or similar image of the patient. Forexample, the image can be a photograph taken during the same orthodonticappointment, from which the modified images were automatically generatedaccording to embodiments disclosed herein, thereby showing the patientthe orthodontic treatment plan during an initial examination.Alternatively, the image 4620 can be a modified image showing thepatient's dentition at the beginning of treatment (or at anotherselected stage), which can look substantially similar or even virtuallyidentical to a photograph or other image of the patient. By presentingthe modified image 4610 showing treatment progress alongside the image4620 of the patient corresponding to another stage of treatment, such asthe start of treatment, the patient can appreciate via the contrastingimages the degree of orthodontic improvement treatment can provide.

FIG. 47 illustrates an augmented display 4700 showing a modified image4710 of the patient in conjunction with a 3D model 4720 of the patient'sdentition. The modified 4710 image and the 3D model 4720 can bedisplayed in adjacent windows or frames of a computer display, forexample. The modified image 4710 can be generated in accordance with theembodiments described herein. For example, the modified image 4710 cancorrespond to the visual depiction 4500 illustrated in FIG. 45 . The 3Dmodel 4720 can be a 3D model of the patient's dentition, based off aselected stage of a treatment plan. In some instances, the 3D model caninclude labels depicting an amount of malocclusion or an amount ofadjustment to be made to one or more portions of the patient'sdentition. For example, in FIG. 47 , the 3D model includes labelsindicating a planned interproximal reduction of 0.5 mm for each of aplurality of interproximal spaces between teeth. Similar features tothose disclosed with FIG. 45 can be included in the augmented display4700, such as a scrollbar or other interface and an optional videoillustration of predicted progress during a treatment plan. For example,the augmented display 4700 illustrates a scrollbar positioned below the3D model, which can be used to adjust the displayed stage of treatmentfor both the modified image and the 3D model, allowing the patient tosee both change simultaneously. Optionally, an image of the patient at afixed stage of treatment, such as the start of treatment, can bedisplayed in a third window, similar to the display illustrated above inFIG. 46 . Optionally, different stages of the 3D model can also be shownsimultaneously in nearby windows, as may be helpful to illustratechanges in the patient's dentition.

FIG. 48 illustrates a treatment-comparison display 4800 for comparing aplurality of treatment plans. A plurality of modified images 4810, 4820,and 4830 of the patient are simultaneously displayed, each showing amodified image according to a different, alternative treatment plan.Each modified image can be generated in accordance with embodimentsdisclosed herein, such as the methods disclosed above in FIGS. 5A-C. Inparticular, the method for generating the virtual depictions of aplurality of alternative orthodontic treatments for a patient describedin FIG. 5C can be employed to generate each of two or more respectivepluralities of modified images, each respective plurality of modifiedimages illustrating the patient's face and dentition (or respectiveportions thereof) in various stages of a respective orthodontictreatment plan. Similar features to those disclosed with FIG. 45 can beincluded in the treatment-comparison display 4800, such as a scrollbaror other interface and an optional video illustration of predictedprogress during a treatment plan. For example, the treatment-comparisondisplay 4800 illustrates a scrollbar positioned below one of themodified images, which can be used to adjust the displayed stage oftreatment for each modified image of the plurality of treatment plans,allowing the patient to see each change simultaneously. Thetreatment-comparison display allows patients, as well as medicalprofessionals, to view the progress and outcome of alternativetreatments in parallel, which can aid in directly perceiving the costsand benefits of different treatment options.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

The invention claimed is:
 1. A computer-implemented method comprising:generating a plurality of 3D models, each of the plurality of 3D modelscomprising a virtual representation of a patient's dentition at adifferent treatment stage of an orthodontic treatment plan; obtaining animage of the patient's face and at least a portion of the patient'sdentition; projecting the obtained image into a 3D space to create aprojected 3D model of the patient's face and the at least the portion ofthe patient's dentition; generating a plurality of modified images ofthe patient, each of the plurality of modified images representing thepatient in a respective treatment stage of the orthodontic treatmentplan, wherein each of the plurality of modified images is formed bycombining the patient's dentition of one of the plurality of 3D modelswith the patient's face of the projected 3D model; receiving, from auser, a selection of a first treatment stage of the orthodontictreatment plan; and sending instructions to display, on a display, afirst modified image corresponding to the selected first treatment stageand a first 3D model corresponding to the selected first treatmentstage.
 2. The computer-implemented method of claim 1, furthercomprising: receiving first identifiers of a first set of referencepoints modeled on the first 3D model, the first set of reference pointscorresponding to a set of anatomical points on the patient's dentitionof the first 3D model; and receiving second identifiers of a second setof reference points represented on the at least the portion of thepatient's dentition included in the obtained image.
 3. Thecomputer-implemented method of claim 2, further comprising: aligning atleast a portion of the first set of reference points on the first 3Dmodel with at least a portion of the second set of reference points onthe projected 3D model of the obtained image.
 4. Thecomputer-implemented method of claim 3, wherein at least one of theplurality of modified images of the patient is generated based on thealignment.
 5. The computer-implemented method of claim 1, furthercomprising sending instructions to display the first modified image in afirst window or frame of the display, and to display the first 3D modelin a second window or frame of the display.
 6. The computer-implementedmethod of claim 1, further comprising: receiving, from the user, aselection of a second treatment stage of the orthodontic treatment plan;and sending instructions to update the display to display a secondmodified image of the patient corresponding to the selected secondtreatment stage and a second 3D model corresponding to the selectedsecond treatment stage.
 7. The computer-implemented method of claim 1,wherein the selected first treatment stage is a final treatment stage ofthe orthodontic treatment plan.
 8. The computer-implemented method ofclaim 1, further comprising: generating a video file that includesrepresentations of the plurality of modified images arranged inchronological order to show the patient's face and the at least theportion of the patient's dentition during the orthodontic treatmentplan.
 9. The computer-implemented method of claim 1, further comprisingsending instructions to display at least one label depicting an amountof adjustment to be made to one or more portions of the dentition of thepatient.
 10. The computer-implemented method of claim 9, wherein the atleast one label identifies one or more of a type of the adjustment or amalocclusion corresponding to the adjustment.
 11. A computer-implementedmethod comprising: generating a first plurality of 3D models, each ofthe first plurality of 3D models comprising a virtual representation ofa patient's dentition at a different treatment stage of a firstorthodontic treatment plan; obtaining an image of the patient's face andat least a portion of the patient's dentition; projecting the obtainedimage into a 3D space to create a projected 3D model of the patient'sface and the at least the portion of the patient's dentition; generatinga first plurality of modified images of the patient, each of the firstplurality of modified images representing the patient in a respectivetreatment stage of the first orthodontic treatment plan, wherein each ofthe first plurality of modified images is formed by combining thepatient's dentition of one of the first plurality of 3D models with thepatient's face of the projected 3D model; and sending instructions todisplay at least one of the first plurality of modified images in afirst window or frame of a display.
 12. The computer-implemented methodof claim 11, further comprising: sending instructions to display theobtained image in a second window or frame of the display.
 13. Thecomputer-implemented method of claim 11, further comprising: sendinginstructions to display at least a second one of the first plurality ofmodified images in a second window or frame of the display.
 14. Thecomputer-implemented method of claim 11, further comprising: sendinginstructions to display a second modified image corresponding to a finaltreatment stage of the first orthodontic treatment plan in a secondwindow or frame of the display.
 15. The computer-implemented method ofclaim 11, further comprising: generating a second plurality of 3Dmodels, each of the second plurality of 3D models comprising a virtualrepresentation of the patient's dentition at a different treatment stageof a second orthodontic treatment plan; and generating a secondplurality of modified images of the patient, each of the secondplurality of modified images representing the patient in a respectivetreatment stage of the second orthodontic treatment plan.
 16. Thecomputer-implemented method of claim 15, wherein each of the secondplurality of modified images is formed by combining the patient'sdentition of one of the second plurality of 3D models with the patient'sface of the projected 3D model.
 17. The computer-implemented method ofclaim 15, further comprising: sending instructions to display at leastone of the second plurality of modified images in a second window orframe of the display.
 18. The computer-implemented method of claim 17,further comprising: receiving, from a user, a selection of a firsttreatment stage of the first orthodontic treatment plan and a selectionof a second treatment stage of the second orthodontic treatment plan;and sending instructions to display, in the first window or frame of thedisplay, a modified image, of the first plurality of modified images,corresponding to the selected first treatment stage and to display, inthe second window or frame of the display, a modified image, of thesecond plurality of modified images, corresponding to the selectedsecond treatment stage.
 19. The computer-implemented method of claim 15,further comprising: generating a video file that includesrepresentations of the first plurality of modified images arranged inchronological order to show the patient's face and the at least theportion of the patient's dentition during the first orthodontictreatment plan and representations of the second plurality of modifiedimages arranged in chronological order to show the patient's face andthe at least the portion of the patient's dentition during the secondorthodontic treatment plan.
 20. The computer-implemented method of claim11, further comprising: receiving first identifiers of a first set ofreference points modeled on a first 3D model of the first plurality of3D models, the first set of reference points corresponding to a set ofanatomical points on the patient's dentition of the first 3D model;receiving second identifiers of a second set of reference pointsrepresented in the at least the portion of the patient's dentitionincluded in the obtained image; and aligning at least a portion of thefirst set of reference points with at least a portion of the second setof reference points.